Zi-Cong Huang

Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States

This work collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N, and factorizes each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states.