Repeated-Token Diagnostics for Transformer Cache Traces
TL;DR
Results show that repeated tokens expose context-conditioned, slice-local similarity in Transformer cache traces, providing an empirical foundation for future repeated-token-aware KV-cache optimization and clarifying representation constraints for such optimizations.
Abstract
Long-context inference in decoder-only transformers is increasingly limited by key–value (KV) cache memory. This work investigates a complementary empirical question: when the same token ID recurs, do its hidden-state and KV traces exhibit coherent structure with low-dimensional variation? We present a repeated-token diagnostic study of Transformer cache traces. We collect hidden-state, pre-RoPE key, and value traces, and compare token-only, token+local-context, and token+position grouping within fixed layer–head slices. On LongBench-v1 traces from Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3, duplicate token IDs account for 96.2–97.8% of hidden-state occurrences. Local-context buckets raise hidden-state centroid cosine to 0.90–0.94, and token+local-context groups have stable rank near 2 and top-4 spectral energy near 0.89–0.95 across hidden, pre-RoPE key, and value traces. These results show that repeated tokens expose context-conditioned, slice-local similarity in Transformer cache traces, providing an empirical foundation for future repeated-token-aware KV-cache optimization and clarifying representation constraints for such optimizations.