AWa: An Activation-Weight Distribution-Aware Binarization Method for Large Language Models
Large Language Models (LLMs) demonstrate outstanding performance across diverse tasks but face significant deployment challenges due to their massive computational and memory requirements. Binarization, representing weights with 1-bit precision, offers a promising solution by dramatically reducing memory footprint and computational costs. However, existing methods often lead to severe performance degradation or require computationally expensive iterative optimization. We propose AWa (Activation-Weight distribution-aware), a novel binarization framework that analyzes the distributions of activations and weights to perform binarization in a systematic, stage-wise manner. First, we perform a joint saliency estimation that determines the saliency of weight by leveraging activation distributions and the Hessian sensitivity of weights. Subsequently, we apply both column and row scaling to the remaining non-salient weights to mitigate quantization errors. By adopting a distribution-based approach that eliminates repetitive operations, our method improves performance while incurring only a marginal increase in binarization time. Extensive experiments on various models and datasets demonstrate that AWa achieves superior zero-shot average accuracy. It surpasses the fastest existing method by a significant margin of 27.3 percentage points on LLaMA-3 70B, while maintaining negligible binarization overhead. Code will be released upon acceptance.