Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.
A unified efficiency analysis is presented showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost, and SparseSpec-L, a training-free self-speculative decoding framework for long-context inference is introduced.
Yue Liu, Yuan Zeng, Min Lyu et al.· arXiv.org· 0 citations
This work proposes a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective that directly targets the expected verified length.
ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions, prevents candidate obsolescence and achieves up to 1.92$\times$ speedup over state-of-the-art multi-candidate methods.
Zhi-Kai Chen, Jun Tao, Weihao Mao et al.· 0 citations
Approximate Speculative Decoding (ASD) is introduced, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection and reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes.
Yuan-Nuo Feng, Zegang Peng, Yu-Xin Xie et al.· 0 citations
This work presents a principled analysis of the distributions induced by lossy verification methods, and shows that many seemingly distinct approaches differ only superficially and can be unified into two categories: truncation-based verification and collaborative verification.
Tianyu Wang, Yuxuan Zhou, Heng Li et al.· arXiv.org· 0 citations
ReTrace is introduced, a rejected-trajectory conditioning method that conditions each draft block on the rejected suffix from the previous round rather than generating it from fresh mask placeholders alone, indicating that the draft model can retain useful semantic and structural information despite local token-level errors.
Luxi Lin, Zhan-Peng Zeng, Shuang Peng et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
