Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match

Dong Li, Yixing Xu, Shuo Yang, Edith Ngai, Emad Barsoum, Jinze Li, Xuanwu Yin, Jinfeng Xu, Guanchen Li

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• AMD-AGI/FLy

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens in parallel from a smaller draft model, yet its strict exact-match verification discards many semantically valid continuations. Moreover, existing training-based SPD methods often suffer from performance degradation on out-of-distribution (OOD) tasks. To this end, we propose Training-Free Loosely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model's self-corrective behavior to judge whether a draft-target mismatch remains semantically valid. FLy introduces a two-tier mechanism: an entropylevel gate that identifies whether the current token allows multiple plausible alternatives or is nearly deterministic, and a token-level deferred window that distinguishes genuine errors from differently worded yet semantically correct variants. To further reduce latency, we design a multi-level acceleration strategy that accelerates not only the target model but also the drafter itself. Owing to its trainingfree design, FLy composes seamlessly with arbitrary draft-target pairs and generalizes across models and domains without hyperparameter re-tuning. Experiments show that FLy preserves ≥99% of the target model's accuracy while achieving an average 2.81× speedup on Llama-3.1-70B-Instruct and 5.07× speedup on the 405B variant. Notably, on out-of-domain datasets, our method remains highly effective and outperforms the training-based method EAGLE-3 by 1.62×. Our code is available at https://github.com/AMD-AGI/FLy.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2511.22972")
get_code_for_paper("2511.22972")
have("2511.22972")

Connect an agent — have() is free.