Switching Linear Attention
Designing expressive sequence layers with efficient inference remains a central challenge in modern machine learning. Standard softmax attention achieves excellent sequence modeling performance through rich nonlinear token interactions, but it requires a key-value cache that grows linearly with sequence length, limitin...
Hyun Dong Lee, X. Gonzalez, Nicolas Zucchet et al.
· 0 citations