HyCo2: Hybrid Context Compression with Global Semantics and Local Evidence

Abstract

Long-context inference is costly and can be brittle for large language models (LLMs): self-attention grows quadratically with input length, and retrieval-augmented generation (RAG) may introduce redundant context that obscures decisive evidence. Existing compressors face a representational trade-off. Hard compression keeps inspectable source tokens but can remove document-level cues needed for disambiguation, whereas soft compression preserves compact semantic summaries but may omit fine-grained facts. We formulate compression as allocating a constrained context budget across these complementary roles and propose Hybrid Context Compression (HyCo2). HyCo2 combines instruction-aware soft tokens from a hybrid adapter with original tokens selected by a lightweight evidence pathway. Ablations support pathway-specific reconstruction and completion pretraining before joint instruction tuning with self-distillation. Across seven knowledge-intensive QA benchmarks and three frozen LLM families, HyCo2 attains QA accuracy comparable to the strongest extractive compressor while using a shorter compressed context, a smaller resident compressor, and lower measured pipeline cost. Relative to uncompressed RAG, it reduces the average decoder-context length by 88.8% and matches or exceeds accuracy on several dataset–backbone pairs. On LongBench, HyCo2 obtains 22.03 under a shared 2,000-token source cap, within 0.29 points of LongLLMLingua with its 7B compressor.1