#artificial intelligence
Sep 2026
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
It is shown that this gating mechanism causes SSMs to first learn an in-weights"memorization"solution, while delaying, or even preventing, convergence to a correct in-context learning solution, and that gating is often beneficial for improving generalization to long sequence lengths.
W. Tong, Aryo Lotfi, Emmanuel Abbé et al.
· arXiv.org · 1 citation