Effective Context Management for LLM Agents via Hybrid Chat History Optimization

Abstract

Large Language Model (LLM)-based agents increasingly power conversational assistants, automation, and decision-support workflows. Their usefulness over long interactions depends on which prior dialogue is carried forward under a fixed context budget. Naive strategies either exhaust the window with full transcripts or compress history so aggressively that salient facts are dropped. We present a hybrid dialogue context management stack that combines a short fixed-size recent-message buffer, rolling summarization of older segments, similarity-based retrieval over embedded utterances and summaries, and a token-budgeted prompt assembler. Unlike prior tiered-memory or static-retrieval pipelines that treat these as separate subsystems, our contribution is a single scoring-and-trimming layer that jointly ranks buffer, summary, and retrieved units under one budget. The design targets scalable, repeatable prompting rather than ad hoc truncation.On a reproducible synthetic multi-turn benchmark (204 user turns per strategy; eighteen recall probes across eight long dialogues), we compare full-history, summarization-only, and hybrid context policies using the same backbone and automated judges. Full-history prompting achieves the highest mean relevance (4.67/5) and the lowest hallucination rate on probes (16.7%) but the largest average prompt size (539 tokens per turn). Summarization-only minimizes tokens (246) but reduces mean relevance to 4.11 and raises probe hallucination to 38.9%. Hybrid balances cost and quality: 31.7% fewer tokens than full history (368 vs. 539) while improving relevance over summarization (4.56) and lowering probe hallucination versus summarization (33.3%). Substring-based context recall is 94.4% across all three policies on the same probes. These results motivate explicit dialogue memory policies as a systems lever for token efficiency without assuming that aggressive compression is free.