The LZ penalty is introduced, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability and without instances of degenerate repetition, and enables state-of-the-art open-source reasoning models to operate with greedy decoding without loss of capability and without instances of degenerate repetition.
Abstract
We introduce the LZ penalty, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability. The penalty is based on the codelengths in the LZ77 universal lossless compression algorithm. Through the lens of the prediction-compression duality, decoding the LZ penalty has the interpretation of sampling from the residual distribution after removing the information that is highly compressible. We demonstrate the LZ penalty enables state-of-the-art open-source reasoning models to operate with greedy (temperature zero) decoding without loss of capability and without instances of degenerate repetition. Both the industry-standard frequency penalty and repetition penalty are ineffective, incurring degenerate repetition rates of up to 4%.
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that r...
Guoxuan Xia, Luka Ribar, Paul Balanca· arXiv.org· 2 citations
Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter, and derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang to support fast verification for models with Gated Delta Net layers.
Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy et al.· arXiv.org· 1 citation
This paper develops a rigorous information-theoretic framework for non-repetitive sequence generation and uses it to provide theoretical underpinnings for temperature-based sampling, and extends it to the practically relevant and novel online setting, where the model’s token distributions evolve and are revealed sequen...
Jin Sima, Nikolaos Papagiannis, Ananth Grama et al.· 0 citations
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicit...
Shikai Qiu, Marc Finzi, Yujia Zheng et al.· arXiv.org· 0 citations
This work generalizes the SPARC construction, and considers the class of \emph{additive orthogonal} regression codes, of which standard SPARCs are a special case, and derives nonasymptotic bounds on the squared-error distortion by tracking the evolution of the encoding residual across stages.
It is argued that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation, and the newly proposed DLM-based framework advances the state of the art in lossless text compression.
Angelo Nardone, P. Ferragina· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.
