Plans Fade Fast: Why Context Management Is Load-Bearing for LLM Agents

Abstract

Context managers summarize or evict old messages so agents can operate beyond a finite window. This creates a reliability problem: removing history that still shapes the model's computation may change later actions, while retaining everything defeats compression. We study plans because agents create them early and are expected to use them later, yet behavioral success alone cannot reveal whether later calls still depend on the plan exchange. We make four contributions. First, we introduce replay pairing, which holds the action-observation trajectory fixed and compares pre-action hidden states with and without the plan exchange. We show that, on Llama-3.1-70B in ALFWorld, the mean contrast across 15 sampled layers decreases 4.1x from the first post-plan action to the second; Llama-3.1-8B shows a similar 5.6x decrease. Second, we show that a simple linear model using layer-32 hidden states predicts later contrast on entirely held-out tasks better than a baseline that only knows the action-call offset (R^2 = 0.847 vs. 0.575). Third, we identify confounds that limit this measurement: the paired histories differ in length, token positions, and discourse structure, and replayed reasoning can reintroduce plan-derived text. Reasoning traces therefore require symmetric controls, and native Qwen3 shows that the Llama timing pattern is not universal. Fourth, in a separate descriptive compression test, we find pooled ALFWorld run success of 56.7% with full history and 22.0% with naive eviction; preserving or re-surfacing only the plan does not recover the gap. Replay pairing measures sensitivity to retaining an exchange, not isolated plan content or later eviction. It gives researchers a concrete audit tool and suggests that context policies should preserve broader task state, not plans alone.