Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention
The Dynamical Memory Transformer is presented, a hybrid architecture that separates sequence encoding from information reading and achieves the best results on several cognitive tasks and language modeling performance close to that of Transformer, but remains behind Mamba in language modeling.
Y. Bendi-Ouis, X. Hinaut
· 0 citations