Sara Rizwan

Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model find...