2026· Annual Meeting of the Association for Computational Linguistics· pp. 19856-19874· 0 citations· 44 references
Computer Science
TL;DR
X-R OUTER is presented, a dual-axis routing framework that separates retrieval necessity from reasoning necessity under a user-defined cost–quality trade-off and reduces QA benchmarks across six QA benchmarks.
Abstract
Large Language Models (LLMs) are often augmented with Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting, yet static “always-on” use is computationally wasteful. Existing adaptive meth-ods typically optimize a single axis, overlooking that evidence need and reasoning depth are only partially correlated. We present X-R OUTER , a dual-axis routing framework that separates retrieval necessity from reasoning necessity under a user-defined cost–quality trade-off. Offline, X-R OUTER profiles four pipelines (D IRECT , RAG, C O T, RAG+C O T) and derives supervision by selecting the utility-maximizing strategy that trades answer quality against token usage and latency. On-line, a compact dual-head router, conditioned on cost weights, uses lightweight probes— retrieval-score dispersion (NQC) and single-pass draft negative log-likelihood (NLL)—to decide whether to invoke RAG and/or CoT without sampling or model internals. Across six QA benchmarks, X-R OUTER reduces
Results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance, and show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
Niqi Lyu, Pengtao Shi, Wei Qiu et al.· 0 citations
The Difficulty-Aware Hybrid Router is proposed, a lightweight framework that jointly leverages query-level, context-level, and draft-level features to classify each request into one of three inference tiers: direct answering by a small model, small-model answering with lightweight verification, or full processing by a large model.
Long-horizon LLM agents require memory systems that recover useful evidence from large interaction histories without passing excessive context to downstream models. Existing memory pipelines often rely on hand-crafted heuristics and repeated LLM calls, which can introduce redundant context and high inference cost. We propose MemoryCPT, an end-to-end trainable agent memory pipeline that spans offline memory construction and online query-conditioned context generation. MemoryCPT consists of two stages: Query-agnostic Distillation (QAD), which distills a modular memory-construction pipeline into a compact model using explicit reasoning traces; and Query-aware Retrieval and Summarization (QAR), which combines reciprocal rank fusion (RRF) with a LoRA-based summarizer trained via Group Relative Policy Optimization (GRPO) under a cost-aware reward. We further introduce Quality per Cost (QPC) to quantify answer quality per unit inference cost. Experiments on LoCoMo and LongMemEval show that MemoryCPT improves the cost-performance trade-off over the evaluated baselines, while ablation and sensitivity analyses characterize the contributions of its components and the effects of key design choices.
Songxin Lei, Kun Ouyang, Weilin Ruan et al.· 0 citations
This work proposes MARS, a scalable knowledge graph question answering (KGQA) approach that requires no model fine-tuning, and performs a structured retrieval procedure that links question entities to the KG and iteratively retrieves relevant next-hop information.
Nikit Srivastava, Daniel Vollmers, René Speck et al.· 0 citations
Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.
Shuoshuo Wang, Fang Xi, Wenyuan Huang et al.· 0 citations
Large Language Models (LLMs) trained using Chain-of-Thought (CoT) supervision have achieved state-of-the-art performance on complex reasoning tasks. However, the generation of long reasoning chains introduces substantial computational overhead during inference, limiting their deployment in low-latency and resource-constrained environments. This paper proposes AdaptiReason, a novel framework that dynamically compresses intermediate reasoning steps based on task difficulty and model confidence without requiring retraining of the underlying base model. AdaptiReason employs a lightweight difficulty estimator to determine the appropriate reasoning depth for each input, followed by a learned token-pruning policy that eliminates redundant or low-information reasoning steps. Experimental evaluation on the MATH, GSM8K, and ARC-Challenge benchmarks demonstrates that AdaptiReason reduces the average number of generated tokens by 3.7× while preserving 98.2% of the baseline reasoning accuracy. Furthermore, the proposed framework is model-agnostic and can be seamlessly integrated with instruction-tuned LLMs without requiring access to model parameters, relying solely on output logits for adaptive reasoning compression. The results demonstrate that AdaptiReason significantly improves inference efficiency while maintaining high reasoning performance, making it suitable for real-time and resource-constrained LLM applications.
V. A, Mithaguru, Amrita Kundu et al.· 2026 4th International Confe...· 0 citations