Codebook-Based Acceleration for Ternary Language Model Inference

Abstract

Emerging 1.58-bit large language models such as BitNet restrict weights to {-1, 0, +1}, dramatically reducing model size. Yet commodity hardware fails to fully exploit this structure: each ternary value carries only log2 (3) ≈ 1.58 bits of information but must occupy a 2-bit container, wasting 21% of memory bandwidth. Furthermore the lack of native mixed-precision support forces costly runtime dequantization. We present a hardware-software co-design that addresses both inefficiencies. On the software side, we compress ternary weights to 1.5 bits per weight via a 64-entry codebook with 6-bit indices, sacrificing only 5.4% of representational capacity; Vector Quantization aware fine-tuning retains 92.6% of baseline accuracy across six benchmarks. On the hardware side, we design TERINT-GEMV, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping. Synthesized using a 28 nm commercial PDK, TERINT-GEMV achieves 99.44 TOPS/W, outperforming recently published BitNet accelerators by up to 7.2× in energy efficiency, and delivers 0.065 J/token with a 2.1× throughput gain over the leading CPU-based ternary runtime at one fifth the power.