Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
This work introduces Grouped Value Attention (GVA), which stores grouped values and reconstructs content keys with a learned linear map at inference, which can be absorbed into the query, eliminating the need to materialize content keys in the intended decode path.