T. Pitassi

Poly-attention: a general scheme for higher-order self-attention

This work defines a vast class of generalizations of self-attention mechanisms, which they are called poly-attention mechanisms, and gives a new attention mechanism which can be computed exactly in quadratic time, and which can perform function composition for any fixed number of functions.