Bin Guo

TideKV: Delayed KV eviction via layer-wise chunked prefill for memory-efficient long-context LLM inference

Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill. Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak...