What Attention Recalls and Recurrence Controls in Hybrid Language Models
Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from o...
Kirill Afendulev, Alexey Dontsov, Elena Tutubalina et al.
· 2 citations
· ⚡1