This work introduces MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed...
This work frames 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly, and presents serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant.
Inesh Chakrabarti, David Limpus, Aditi Rana et al.· arXiv.org· 0 citations
This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusi...
Wenzong Yang, Danyang Zhang, Kunteng Cao et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.