Preprint
Aug 2026
vToken: Token-Level Virtualization for Reclaimable KV Caches
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch...
Yuanhang Gao, Xiangrui Yang, Yuan-Feng Chen et al.
· 1 citation