BitNet v2: Low-bit and Sparsified Activations for 1-bit Large Language Models.

Abstract

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which pose significant challenges to low-bit quantization. In this work, we investigate sparse and low-bit activation strategies tailored for 1-bit LLMs. We first introduce BitNet v2 a4.8, a hybrid quantization-sparsification architecture designed to mitigate quantization errors arising from outlier activation channels. BitNet v2 a4.8 utilizes 4-bit quantization for inputs to attention and feed-forward layers, while sparsifying intermediate activations and subsequently quantizing them to 8 bits. To further leverage the 4-bit computation capabilities of next-generation GPUs, we present BitNet v2, a novel framework that enables fully 4-bit activations for 1-bit LLMs. To address the outlier activation channels, we propose $/mathcal{H}$, a lightweight module that applies an online Hadamard transformation prior to quantization, effectively regularizing activation distributions into more Gaussian-like forms conducive to low-bit representation. Extensive experiments demonstrate that BitNet v2 a4.8 achieves performance comparable to BitNet b1.58 with similar training costs. Furthermore, BitNet v2, when trained from scratch with 8-bit activations, matches the performance of BitNet b1.58. Remarkably, training BitNet v2 directly with native 4-bit activations leads to only marginal performance degradation, while significantly reducing computational overhead during batched inference.