Prashanna Mani Paudel

Linger and Lose: Knowledge Collapse in Low-Bit Language Models

Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographi...