MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

Ashitabh Misra Madhav Agrawal Arham Jain Tarek F. Abdelzaher

Jul 2026 · arXiv.org · Vol abs/2607.23047 · 0 citations · 37 references

Computer Science

TL;DR

It is shown that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation, and proposed MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer, outperforms adaptive and mixed-precision baselines in every setting.

Abstract

Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget. In practice the budget varies across deployments and is unknown at calibration time. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers. We show that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation. We propose MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer. MixQuant marginalizes each layer's distortion over random quantized upstream configurations to obtain budget-agnostic scores, calibrates the quantizer's parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths. A single greedy pass then serves any budget at deployment. Across Llama-3.2-3B, Llama-2-7B, and Mistral-7B under AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 at the tightest budget, while matching an ILP solver at negligible deployment cost.

DOI arXiv Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Aug 2026

Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs

Q-Strata is proposed, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model, achieving lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ i...

Deokjae Lee, Si-Hun Chu, Hyun Oh Song · 0 citations

#machine learning Preprint Aug 2026

Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space

This work proposes code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performs guided search to preserve deployment faithfulness in fine-tuning low-bit models across different quantization datatypes.

Shiguang Wu, Zhou-Chen Lin, Quan-Ming Yao · 1 citation

#machine learning Preprint Sep 2026

The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally

Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground...

Jun-Hao Hu, S. Ramachandran · 1 citation

Jul 2026

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

This work introduces MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT, and establishes a new balance between accuracy and resource efficiency for LLM quantization.

Simla Burcu Harma, D. Mishin, Zheng-Yuan Su et al. · arXiv.org · 1 citation

Preprint Aug 2026

SoftWater: Class-Aware Rate Allocation for Softmax Quantization

Across five models from 1B to 32B, SoftWater outperforms the released WaterSIC quantizer at matched head rates on 59 of 60 test points, using none of that pipeline's refinements and cutting head-induced KL by $6.5\times$--$8.3\times at 2 bits.

Joao V. Cavalcanti, Ashia C. Wilson · 2 citations

Preprint Aug 2026

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

This work systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks and establishes that quantization degradation is governed by how errors are introduced at the source and how they accumulate across the network.

Chen-Xi Zhou, Pengfei Cao, Jin Ye et al. · 0 citations