Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parame...
Florian Eichin, Philipp Mondorf, Andrei Mircea et al.
· 0 citations