It is demonstrated that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.
Abstract
How does data shape language model (LM) behavior throughout pretraining? We investigate this question through a case study on entity comparison, e.g., Between France and Brazil, which country is larger?, across nine comparison tasks spanning five entity types. We begin with controlled experiments in which we train small LMs (163M parameters) on mixtures of natural text from pretraining corpora and synthetic task data. Our analysis identifies three distinct phases of learning: (1) an early phase where the LM selects entities by frequency, (2) a middle phase where the LM selects entities by position in the prompt (first vs. last mentioned entity), and (3) a late phase where the LM selects the entity that is the correct answer. We show that the emergence of these three phases is causally determined by statistical properties of the training data. With small amounts of task-specific synthetic data, we observe only the first two phases and the model fails to learn the task; with large amounts, the model jumps directly from the frequency-based heuristic to solving the task correctly. Moreover, changing the entity frequency distribution from a naturally occurring Zipfian distribution (a small number of entities are very common and the vast majority are rare) to a uniform one weakens the frequency phase and speeds up learning of the task. Lastly, we show that the three phases of training dynamics from our controlled experiments appear in the open source model OLMo-2 32B. Together, our findings demonstrate that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.
The results show that a model's inductive generalization reflects the latent organization of its training data, with implications for data curation and for predicting how far a learned fact will spread.
Jan Sobotka, Ekaterina Goliakova, Marcin Detyniecki et al.· 0 citations
Adapting LLMs to specialized domains critically depends on high-quality training data, making dataset construction a decisive factor in domain performance. This survey reviews methods for constructing domain-adaptation training datasets through a structured, data-centric analysis. Based on a comparative analysis of rep...
Xin-Yuan Qiu, Ke-Lan Ren, Xiao-Long Cui et al.· Journal of King Saud Univers...· 0 citations
This work reviews data strategies across the training lifecycle and formulates them as policies that allocate a training budget over candidate examples, then divides methods by their decision granularity, model dependence, and temporal adaptivity, then compares their computational requirements and empirical support.
Yuan-Jian Xu, Jia-Ning Hao, Tian-Ze Sun et al.· 0 citations
Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchma...
Despite remarkable progress in language understanding and generation, decoder language models continue to exhibit striking failures on numerical data. Numerical examples are typically serialized as text by concatenating feature names with their values (e.g., fat:3.4), while conventional task adaptation applies loss onl...
Weimin Lyu, Jing-Feng Zhang, Ling-Dao Sha et al.· 0 citations
Entity matching is the task of deciding whether two records describe the same real-world entity. Current entity matching methods rely on pre-trained language models (PLMs) which require task specific fine-tuning or generative large language models (LLMs) which do not require task-specific fine-tuning but are slow and e...
Aaron Steiner, Ralph Peeters, Christian Bizer· 0 citations
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.