Entropy-Guided LLM Decoding via Probabilistic Circuits

Zhizhen Chen Daniel Mingyi Israel Guy Van den Broeck Zhe Zeng Kareem Ahmed Sameer Singh Entropy-aligned K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano

· 0 citations · 16 references

TL;DR

This work argues that future entropy is also essential for LLM decoding, and estimates its lower and upper bounds by computing the lower and upper bounds on a hidden Markov model (HMM), which is distilled as a surrogate of the LLM.

Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

This work introduces Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning and devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing...

Dun-Yao Xue, Cheng-Shuo Du, Zheng-Bo Wang et al. · 0 citations

Preprint Sep 2026

Approximate maximum-likelihood decoding via truncated free energies

Maximum-likelihood decoding (MLD) achieves the minimum logical error rate of stabilizer codes under known i.i.d. Pauli noise, but its exact evaluation is \#P-hard. Practical pipelines therefore approximate MLD by minimum-weight decoding (MWD), retaining only the lowest-weight recovery per syndrome and discarding the co...

Yuan-Qi Liu, Wei-Lei Zeng, Jun-Jie Wu et al. · 0 citations

Preprint Aug 2026

LLM-Assisted LDPC Decoding via Syndrome-Verified Semantic Priors

Semantic communication exploits the meaning of the payload, which bit-level processing discards. When channel decoding fails on a natural language payload, the errors appear as corrupted characters in the recovered text. A large language model (LLM) infers the intended characters from the semantic context, but it can a...

Sojeong Park, Hyeonsu Lyu, Jaehyun Choi et al. · 1 citation

2026

Gaussian-Elimination-Free BP-OSD Algorithm for Short 5G LDPC Codes

In this paper, we propose a two-stage belief-propagation ordered statistics decoding (BP-OSD)-like algorithm, hereafter referred to as BP-LCGCD, tailored to 5G low-density parity-check (LDPC) codes. When standard BP decoding fails, a modified BP (mBP) with an iteration limit determined by the code’s girth and a normali...

Qian-Fan Wang, Yi-Wen Wang, Jia-Yi Yang et al. · IEEE Transactions on Communi... · 0 citations

Preprint Aug 2026

Logical Neural Belief Propagation for Linear-Complexity Decoding of Surface Codes

Logical Neural Belief Propagation (L-NBP) is proposed, a BP-based neural decoder that redirects the decoding objective from physical-level to logical-level decoding and produces posterior beliefs that are favorable for logical classification.

Heeyoul Kwak, Seong-Joon Park, Dae-Young Yun et al. · 1 citation

#artificial intelligence Preprint Sep 2026

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

An empirical error-propagation analysis is developed and finds that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates.

Gao-Yuan Du, A. Khan, Rex Zhou et al. · 1 citation