Skip to content

Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

Sep 2026 · 0 citations · 33 references
Computer Science

TL;DR

This work introduces Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning and devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing its theoretical approximation guarantees.

Abstract

We introduce Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning. Existing selection strategies rely predominantly on scalar probabilities, ignoring geometric semantic relationships and causing candidate redundancy. Meanwhile, current geometry-aware methods often require complex optimization or directly reweighting the original token probabilities, leading to significant computational overhead or inference instability. To address this, we formulate decoding as a subset optimization problem using a Mahalanobis distance-driven objective to enhance semantic diversity while preserving high probabilities. Specifically, we dynamically discount redundant generation paths using a token similarity matrix, constructed via an adaptive-bandwidth kernel over token embeddings. We further devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing its theoretical approximation guarantees. This renders ME-Decoding a robust, plug-and-play module with negligible inference overhead. Extensive experiments across diverse reasoning and generation tasks demonstrate that our method consistently achieves strong performance.

View source

Similar papers

#machine learning Preprint Aug 2026

Correlation-Aware Structured Pruning for Large Language Models

This work forms the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error, and develops a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection.

Si-Cheng Xu, Hao Shi, Wei Zhang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

An empirical error-propagation analysis is developed and finds that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates.

Gao-Yuan Du, A. Khan, Rex Zhou et al. · 1 citation
#machine learning Preprint Sep 2026

Acceptance-Aware Draft Model Training for Speculative Decoding

This work proposes acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window and shows that directly optimizing the acceptance objective is more effective than conventional distribution-matching objectives.

Tian-Hua Xia, M. Ganesan, Yi-Fei Feng et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling

Diffusion large language models (dLLMs) offer a promising parallel decoding paradigm as an alternative to autoregressive generation through iterative unmasking. However, dLLMs typically require many steps before token confidence reaches the decoding threshold, resulting in inefficient inference even with block-wise KV...

Li-Xuan Wei, Wei Zhou, Jian-Wen Wu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.