C-PTQ is proposed, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error and achieves state-of-the-art performance without auxiliary modules like LoRA, thereby maintaining high efficiency.
Abstract
Multimodal large language models (MLLMs) require huge memory and computational costs, which limits their practical deployment. Post-training quantization (PTQ) techniques offer an efficient solution for model compression and inference acceleration. Yet, the quantized model faces performance degradation due to outlier channels, which are highly sensitive to quantization and substantially impair activation fidelity and task accuracy. To protect these salient channels during quantization, existing PTQ methods leverage modality- or token-level metrics to guide channel-wise scaling (CWS) of LLM decoders. However, these orthogonal measurements fail to capture channel-wise impacts on task-specific loss, and the misalignment between importance and scaling factors ultimately leads to suboptimal performance. To address this issue, we propose C-PTQ, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error. Motivated by second-order derivatives, we design a Fisher-weighted objective as a tractable Hessian approximation, seamlessly injecting task sensitivity into the scaling process. Notably, we achieve state-of-the-art performance without auxiliary modules like LoRA, thereby maintaining high efficiency. Experiments on Qwen2.5VL, InternVL2 and LLaVA-OV across 8 benchmarks demonstrate our effectiveness in both weight-only and weight-activation settings.
ACBQ is presented, a simple yet effective framework that simultaneously addresses weight–activation joint quantization and extreme low-bit weight quantization and an adaptive cross-block quantization strategy that explicitly accounts for cross-layer dependencies by encouraging consistency across blocks.
Hailing Wang, Jianglin Lu, Yitian Zhang et al.· Annual Meeting of the Associ...· 0 citations
KronQ, a PTQ framework that challenges the assumption that all output channels contribute equally to the layer-wise reconstruction objective by introducing the gradient covariance into the quantization pipeline, and introduces bidirectional incoherence processing.
Donghyun Lee, Yuhang Li, Ruokai Yin et al.· 0 citations
This paper introduces Gradient Enhancement Task Aware Post-training Quantization, i.e., GTAQ, to address the generalization issue of Large Language Models, and extensively evaluates the LLaMA family of language models on WikiText, C4, and MMLU.
Yihua Shao, Yangyang Gu, Minxi Yan et al.· 0 citations
Efficient Tuning Before Quantization (ETBQ) is proposed, a pre-conditioning tuning stage for Stochastic Gradient Descent (SGD)-optimized models before PTQ, which improves low-bit PTQ across diverse tasks.
Peng Xia, Junbiao Pang, Muhammad Ayub Sabir· 0 citations
Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.
Large-scale pre-trained Vision-Language Models (VLMs) have demonstrated remarkable performance across various visual and multimodal tasks. However, deploying these models on downstream application platforms remains challenging due to computational demands and domain gaps. Quantization offers a promising solution by significantly reducing these costs, making VLMs more feasible for deployment in such environments. There are two prevailing paradigms: Quantization-Aware Training (QAT), which preserves model performance but incurs substantial training costs; and Post-Training Quantization (PTQ), which offers greater efficiency but introduces multimodal gaps and leads to performance degradation on downstream tasks. To reduce computational costs and bridge domain gaps, we propose the “Prompt for Quantization” (P4Q) by integrating PTQ with Parameter-Efficient Fine-Tuning (PEFT) techniques. P4Q compresses model parameters and activations via PTQ, introducing learnable prompts and low-bit adapters to enhance performance on downstream tasks. The learnable prompts embed downstream knowledge to mitigate domain gaps, while the low-bit adapters realign the distributions of image and text features, thereby mitigating multimodal gaps. We also introduce a distillation loss based on cosine similarity predictions to distill the quantized model using a full-precision teacher model. Extensive experiments on thirteen datasets demonstrate that P4Q significantly enhances the performance of low-bit CLIP while reducing deployment costs. For instance, an 8-bit P4Q compressed CLIP-ViT/B-32 achieves 66.94% Top-1 accuracy on ImageNet, surpassing the prompt fine-tuned full-precision counterpart by 2.24% while reducing model size by 4 \(\times\) . The source code is publicly available at https://github.com/HuixinSun/P4Q_official.
H. Sun, Runqi Wang, Yanjing Li et al.· ACM Transactions on Multimed...· 0 citations