On-Policy Attention Linearization
On-Policy Attention Linearization (OPAL) is introduced in which the hybrid attention student samples its own long-context trajectories and receives dense supervision from the frozen full-attention teacher.
Arian Raje, Anupam Nayak, Anthony Fei et al.
· 0 citations