LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each atten...
Teresa Delgado, Hugo Hazard, Martin Benfeghoul et al.
· 0 citations