Closing the Operational Gap in Semantic Caching
Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor de...
Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev et al.
· 1 citation