All You Need Is Non-Commutative Words
It is shown that the noncommutativity of matrix product captures word order without positional encodings (PEs) and yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost.
Carla Quispe Flores, Stanley Salvatierra, Renan Cabrera
· 0 citations