Temperature Sampling is Entropy-Optimal: An Information-Theoretic Framework for LLM Decoding
This paper develops a rigorous information-theoretic framework for non-repetitive sequence generation and uses it to provide theoretical underpinnings for temperature-based sampling, and extends it to the practically relevant and novel online setting, where the model’s token distributions evolve and are revealed sequen...