Modern LLM workflows, such as retrieval-augmented generation (RAG) and LLM agents, build long prompts from multiple independent text segments, making prefill a major source of computation. Reusing KV caches is therefore important for reducing serving costs and latency: distributed caching systems spread KV cache across...
Yi-Fei Liu, Jun-Bin Pan, Jun-Hao Hu et al.· 0 citations
RedKnot establishes a new foundation for AI infrastructure by transforming the KV cache from a monolithic, passive runtime artifact into a dynamic, model-aware runtime substrate for scalable LLM serving.
Yang Liu, Zhao-Kai Luo, Hua-Yi Jin et al.· arXiv.org· 7 citations· ⚡1
This work presents a DeepSeek-V4 realization of RedKnot's head-aware reuse principle, a DeepSeek-V4 realization of RedKnot's head-aware reuse principle, and describes the factorization, position repair, token-row closure, sparse-MoE support, TP8 integration, and the measurement boundaries needed to interpret these resu...
Yang Liu, Zhao-Kai Luo, Hua-Yi Jin et al.· 0 citations
ExFold is proposed, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode that calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts.