Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
An empirical error-propagation analysis is developed and finds that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates.
Gao-Yuan Du, A. Khan, Rex Zhou et al.
· 0 citations