In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL

Abstract

The sudden emergence of induction circuits and the ability to perform in-context copying in LLMs coincides with a sharp drop in training loss over a short span of the training process. This co-occurrence has been cited as evidence that it marks the point at which a wide range of in-context learning (ICL) capabilities begin to emerge. To analyze the training dynamics of few-shot abstractive ICL capabilities (i.e., tasks where the answer is not contained in the input context), we propose Hapax, a training regime that omits the loss contribution of tokens predictable by the induction circuit to control training pressure for copying. Using this intervention, we find that abstractive ICL capabilities can still emerge, despite a substantial reduction in inductive copying and the absence of the characteristic phase changes observed during Vanilla training. Analysis of attention heads shows that Hapax-trained models develop fewer and weaker induction heads. Together, these results suggest that the developmental link between emergent phenomena associated with induction circuits and the emergence of abstractive ICL is weaker than previously hypothesized. More broadly, Hapax provides a controlled framework for analyzing how copying-related training signals shape the development of in-context learning.