Fast and Accurate Fisher-Guided Quantization via Efficient Kronecker Factorization
TL;DR
This work proposes an efficient Kronecker-factored approximation yielding state-of-the-art performance when integrated into existing quantization schemes, and substantially accelerates the most expensive component in second-order quantization – Hessian parameterization – achieving up to a 10× speedup over prior approaches.
Abstract
Quantization has shown strong results in preserving model quality under compression. However, under aggressive bit-width reductions, even quantization may require additional information to prevent performance degradation. A natural source of it is the second-order curvature information, captured by the Hessian. Since the Hessian of the model layers is pro-hibitively large, direct computation is infeasible, making structured parameterizations and approximations crucial in practice. In this work, we propose an efficient Kronecker-factored approximation yielding state-of-the-art performance when integrated into existing quantization schemes. Evaluations on the LLaMA and Qwen model families show near-baseline quality at 4-bit compression and only a 5–6% degradation at 2-bit for models with 7–8B parameters. Moreover, our method substantially accelerates the most expensive component in second-order quantization – Hessian parameterization – achieving up to a 10× speedup over prior approaches. Quantized model checkpoints are available at https://huggingface.co/collections/ timo13113/fastkron-collection .