Signed Rescue Routing is introduced, a budgeted routing method that predicts these two events separately and ranks requests by their difference and shows that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.
Abstract
Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.
No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-...
Norah Alballa, Wen-Xuan Zhang, Salma Kharrat et al.· 0 citations
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-c...
Sam Wang, Julia White, Sahibzada Allahyar et al.· 1 citation· ⚡1
SeLMRoute is introduced, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives and achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level represent...
RLCascadeRouter is a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop''and model selection, and uses trajectory returns and advantages to directly optimize the performance-cost objective.
Shihong Huang, Sheng-Jie Wang, Hong-Yao Ma et al.· 1 citation
Existing Large Language Model (LLM) routing methods score LLMs independently to select top-$k$ models. However, this ignores model correlations and enforces a rigid computational budget. Consequently, routers often select redundant models that share failure modes, limiting the overall probability of success. To address...
Wang Wei, Harry Yang, Tiankai Yang et al.· arXiv.org· 3 citations
DABO is proposed, a calibration-aware binary offloading method for collaborative large–small model inference that maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge.
Chen Zhu, Yi-Ming Su, Chenwenjie Mao et al.· IEEE Access· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.