A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
The large KV-cache size of modern LLMs creates a barrier to efficient deployment. Recent work has explored replacing attention layers'RoPE positional embeddings with alternative decay-based mechanisms, which can then be used to prune KV-cache during inference. However, these decay functions have limited expressivity, a...
Davis Wertheimer, Hao-Chen Shen, Ahan Gupta et al.
· 0 citations