F. Wei

BitNet v2: Low-bit and Sparsified Activations for 1-bit Large Language Models.

This work introduces BitNet v2 a4.8, a hybrid quantization-sparsification architecture designed to mitigate quantization errors arising from outlier activation channels, and proposes a lightweight module that applies an online Hadamard transformation prior to quantization, effectively regularizing activation distributi...