It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
This study analyzes the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it and suggests that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs.
Raito Kiya, Satoki Ohashi, Kosuke Sato et al.
· 0 citations