KronQ, a PTQ framework that challenges the assumption that all output channels contribute equally to the layer-wise reconstruction objective by introducing the gradient covariance into the quantization pipeline, and introduces bidirectional incoherence processing.
Abstract
Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Most existing second-order PTQ methods, including GPTQ, construct quantization objectives from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), \KronQ{} achieves 7.93 perplexity.
We accelerate a family of algorithms for neural network quantization whose geometry is informed by any Kronecker-factored approximation of the Hessian. GPTQ-style adaptive rounding typically uses one-sided information derived from input activations. Two-sided Kronecker-factored Hessian approximations can additionally capture correlations across output coordinates, but applying GPTQ directly in the vectorized weight domain is computationally expensive. Building on the two-sided adaptive-rounding formulation used by BoA and YAQA, we introduce BaKron, an efficient solver that combines anti-diagonal parallelism with a recursive divide-and-conquer construction. For an $m\times n$ weight matrix, BaKron uses $O(m+n)$ sequential steps while reducing the total work from $O(m^2n^2)$ to $O(mn(m+n))$. Thus, it matches the cubic scaling of GPTQ while exploiting richer curvature information. Moreover, BaKron is modular with respect to both the base quantizer and the Hessian estimator. We also provide practical benchmarks, consider a range of Hessians that BaKron can be called with, find an efficient technique to compute these Hessians, and evaluate the algorithm experimentally.
ACBQ is presented, a simple yet effective framework that simultaneously addresses weight–activation joint quantization and extreme low-bit weight quantization and an adaptive cross-block quantization strategy that explicitly accounts for cross-layer dependencies by encouraging consistency across blocks.
Hailing Wang, Jianglin Lu, Yitian Zhang et al.· Annual Meeting of the Associ...· 0 citations
Quantization has shown strong results in preserving model quality under compression. However, under aggressive bit-width reductions, even quantization may require additional information to prevent performance degradation. A natural source of it is the second-order curvature information, captured by the Hessian. Since the Hessian of the model layers is pro-hibitively large, direct computation is infeasible, making structured parameterizations and approximations crucial in practice. In this work, we propose an efficient Kronecker-factored approximation yielding state-of-the-art performance when integrated into existing quantization schemes. Evaluations on the LLaMA and Qwen model families show near-baseline quality at 4-bit compression and only a 5–6% degradation at 2-bit for models with 7–8B parameters. Moreover, our method substantially accelerates the most expensive component in second-order quantization – Hessian parameterization – achieving up to a 10× speedup over prior approaches. Quantized model checkpoints are available at https://huggingface.co/collections/ timo13113/fastkron-collection .
Viktoriia Chekalina, Gerasin Timofey, Andrey Kuznetsov et al.· Annual Meeting of the Associ...· 0 citations
SCHUROPT is introduced, which analytically eliminates the suffix's optimal continuous response, yielding an exact groupwise quadratic with Schur-complement curvature, and achieves the highest mean zero-shot accuracy among the evaluated backpropagation free PTQ baselines.
Gunjun Lee, Sehwan Son, Younjoo Lee et al.· 0 citations
RDQ (Residual Distribution Quantization), a PTQ framework whose central contribution is Cascaded Error Compensation, a sequential calibration procedure that captures the actual drifted activations each layer receives and fits per-channel AWQ-style scales against those drifted inputs, with scales folded into preceding RMSNorm weights for exact mathematical equivalence at zero inference overhead.
Uniform INT8 quantization is attractive for deploying learned image coding (LIC), but its rate--distortion (R--D) performance is often limited by heavy-tailed tensors and large inter-channel variations. Existing methods mainly adapt the quantizer through mixed precision or non-uniform codebooks. We propose Hadamard-Transform-domain Quantization (HaTQ), which uses orthogonal Hadamard reparameterization before quantization to redistribute weight and activation responses in the original domain across channels. The reparameterization preserves the original function mapping of each linear operator, while making its weights and activations more amenable to uniform INT8 quantization. HaTQ provides two complementary forms. Double-Hadamard quantization transforms both the input activations and weights, whereas weight-only Hadamard quantization transforms only the weights. This distinction is important because the constant Hadamard basis can coherently accumulate a nonzero channel mean and enlarge the activation range in sensitive layers. We identify these sensitive layers through offline profiling and assign the appropriate form to each layer without input-dependent branching. HaTQ supports both post-training quantization (PTQ) and quantization-aware training (QAT), uses uniform INT8 quantizers, and is compatible with integer-only execution. Experiments on representative LIC architectures and datasets demonstrate consistent improvements across different quantization settings. The resulting QAT models further outperform competing mixed-precision and non-uniform quantization methods. TensorRT deployment results demonstrate practical INT8 inference efficiency. The source code will be publicly released.
Junqi Shi, Chongzhi Wang, Yiwen He et al.· 0 citations