Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing me...
Ruo-Ling Qi, Yi-Rui Liu, Xuan'er Wu et al.· 0 citations
SuffixReplay is presented, the first prefix caching system that lets hybrid LLMs reuse cached prefixes at every cache-supported page boundary without materializing recurrent-state checkpoints and is evaluated on three hybrid LLMs.
Yi-Rui Liu, Ruo-Ling Qi, Xuan'er Wu et al.· 0 citations
FlashDrive is proposed, an algorithm-system co-design framework that targets all four stages of Vision-Language-Action inference simultaneously and moves end-to-end autonomous driving substantially closer to real-time deployment.
Ze-Kai Li, Yihao Liang, Hong-Fei Zhang et al.· 2 citations
Tail-Replay is presented, a prefix caching mechanism that enables unconstrained token-level prefix reuse in hybrid large language models and is evaluated on three Gated DeltaNet-based hybrid models using the LongBench and RULER benchmarks.
Yi-Rui Liu, Ruo-Ling Qi, Xuan'er Wu et al.· 1 citation
LLM serving is increasingly accelerated by position-independent caching (PIC). Existing PIC methods, however, are built for full-attention models, where a token-indexed KV cache underlies its core operations: matching reusable token chunks, concatenating their KV entries, and selectively recomputing a few tokens to res...
Yi-Rui Liu, Ruoling Qi, Long-Wen Wang et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
