2026· EPJ Web of Conferences· 0 citations· 6 references
TL;DR
This survey argues that context injection strategy, rather than context capacity, is the defining research challenge for long-context LLM deployment, and proposes a three-axis analytical framework revealing that injection performance is jointly governed by selection, representation, and scheduling.
Abstract
The performance of large language models (LLMs) is fundamentally shaped by what information reaches them at inference time. As context windows expanded, the challenge shifted from can we fit the data to how do we fit the right data, efficiently and faithfully. This survey argues that context injection strategy, rather than context capacity, is the defining research challenge for long-context LLM deployment. We organize the literature into five major families - direct injection, compression-based injection, retrieval-based injection, memory-based injection, and hybrid/agentic injection. Our primary contribution is a three-axis analytical framework revealing that injection performance is jointly governed by selection (what information is chosen), representation (how it is encoded), and scheduling (when it is delivered during inference). Systematic analysis of the surveyed systems shows that current approaches overwhelmingly optimize the selection axis while representation and scheduling remain largely unexplored. We further identify six structural open challenges that are unlikely to be resolved within any single paradigm, and discuss evaluation gaps that current benchmarks fail to address. These findings suggest that progress on representation and scheduling axes, rather than further scaling of context windows, is the most promising direction for closing the gap between nominal and effective context capacity in deployed LLM systems.
As the context window of Large Language Models (LLMs) continues to expand, the data required to effectively train and evaluate these capabilities remains underexplored. With existing research primarily focuses on architectural optimization, there is a need for a systematic, data-centric review. This survey bridges this gap by investigating the data foundations of Long-Context Language Models (LCMs). We begin by examining current data strategies alongside their strengths and limitations, mapping the required data to desired model capabilities. Building on this, we explore how targeted training data designs drive core, often interconnected skills such as retrieval, reasoning, and aggregation. Furthermore, we analyze the evaluation landscape, illustrating how selecting appropriate benchmarks is crucial for probing capability boundaries and guiding effective model selection. Finally, we synthesize actionable guidelines for data construction and outline critical future directions to propel the advancement of long-context language models, including quantifying data quality, establishing scaling laws for length distributions, and developing dynamic evaluation frameworks.
Zechen Sun, Yu-Yang Sun, Zhao-yu Su et al.· Transactions of the Associat...· 0 citations
PredicateLongBench is proposed, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints drawn from a broader predicate class.
As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.
Jinghan Tan, Yuanzheng Wang, Lu Chen et al.· 0 citations
A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include cardinality estimation, set relationships, and grouped statistics, which widely exist in logs, program outputs, tables, and multi-turn conversations. To provide the aggregation state required by these tasks, we introduce a model-side aggregation interface that maintains compact Hash-based HyperLogLog (HLL) sketch states alongside a frozen language model. While the model processes the context, an extractor maps each relevant record to a canonical identity. The identity is then hashed and updates the HLL state. These states can be merged across context segments and/or read out directly for downstream reasoning, avoiding an additional generate-execute-return cycle. We validate the proposed approach by setting the HLL state size as 2 KiB (2,048 registers), which does not increase with context length or set cardinality. In a distinct-count experiment involving one million records, the mean relative error was 1.6%. In a separate merge test, states built from as many as 256 segments produced exactly the same readout as a single pass over the same stream. On 3,969 aggregate-then-reason tasks from 174 source windows, the fixed-budget interface reached 99.2% accuracy on Gemma 4 (31B, BF16), compared with 100.0% under exact aggregation; the paired gap was 0.8 percentage points (95% window-cluster CI: 0.5-1.3 points). On a matched set of 174 items, our method improved over direct full-context reasoning by 63.2 points on Qwen and 56.3 points on Gemma. The corresponding gains over chain-of-thought (CoT) reasoning were 60.9 and 63.2 points, respectively. On a fixed 1,200-task Oolong-Synth subset, our method reached 91.1% on Qwen and 99.3% on Gemma. Code is available at https://github.com/songdc98/sketchops.
Dachuan Song, Ju Yin, Zechen Hu et al.· 0 citations
It is argued that context assembly in ContextPipe is structurally isomorphic to query execution in a relational database: both execute under a hard budget, exploit a tiered cache, and leverage statistics.
A simple method, Self-Guided TTT (S-TTT), which improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
Xinyu Zhu, Zhenqin Xu, Xiaohan Wei et al.· 0 citations