TL;DR
Five self-speculative decoding techniques are characterized across three model sizes and three datasets and recommendations for future research in this area are provided.
A unified efficiency analysis is presented showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost, and SparseSpec-L, a training-free self-speculative decoding framework for long-context inference is introduced...
It is found that the speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification by appending a trained soft prompt at the end of the target sequence, which can repurpose the speculative-decoding module into a sequence classifier.
DFly is proposed, a block-diffusion framework combining a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head, improving target-feature utilization and intra-block dependency modeling while keeping generation parallel, and DFly treats verification as a shared batch-level resource.
This work proposes CURE, a budget-aware dynamic repair tree designed to repair errors at uncertainty focal points without incurring prohibitive tree-verification overheads, and provides a plug-and-play repair module compatible with standard parallel drafting frameworks.
This work proposes a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective that directly targets the expected verified length.
A from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop, finding the best configuration reaches a measured wall-clock speedup.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
