Skip to content

Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

Nov 2026 · 1 citation · 47 references
Computer Science

TL;DR

It is demonstrated that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.

Abstract

How does data shape language model (LM) behavior throughout pretraining? We investigate this question through a case study on entity comparison, e.g., Between France and Brazil, which country is larger?, across nine comparison tasks spanning five entity types. We begin with controlled experiments in which we train small LMs (163M parameters) on mixtures of natural text from pretraining corpora and synthetic task data. Our analysis identifies three distinct phases of learning: (1) an early phase where the LM selects entities by frequency, (2) a middle phase where the LM selects entities by position in the prompt (first vs. last mentioned entity), and (3) a late phase where the LM selects the entity that is the correct answer. We show that the emergence of these three phases is causally determined by statistical properties of the training data. With small amounts of task-specific synthetic data, we observe only the first two phases and the model fails to learn the task; with large amounts, the model jumps directly from the frequency-based heuristic to solving the task correctly. Moreover, changing the entity frequency distribution from a naturally occurring Zipfian distribution (a small number of entities are very common and the vast majority are rare) to a uniform one weakens the frequency phase and speeds up learning of the task. Lastly, we show that the three phases of training dynamics from our controlled experiments appear in the open source model OLMo-2 32B. Together, our findings demonstrate that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.

View source

Similar papers

Review Open access Oct 2026

Training datasets tailored for domain-specific large language models: challenges, methods and trends

Adapting LLMs to specialized domains critically depends on high-quality training data, making dataset construction a decisive factor in domain performance. This survey reviews methods for constructing domain-adaptation training datasets through a structured, data-centric analysis. Based on a comparative analysis of rep...

Xin-Yuan Qiu, Ke-Lan Ren, Xiao-Long Cui et al. · 0 citations

Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

This work reviews data strategies across the training lifecycle and formulates them as policies that allocate a training budget over candidate examples, then divides methods by their decision granularity, model dependence, and temporal adaptivity, then compares their computational requirements and empirical support.

Yuan-Jian Xu, Jia-Ning Hao, Tian-Ze Sun et al. · 0 citations
#natural language process... Preprint Nov 2026

Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models

Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchma...

Aarne Talman, S. Chatzikyriakidis · 0 citations
#artificial intelligence Preprint Oct 2026

MoMaTune: Model, Match, Then Tune Language Models for Numerical Records

Despite remarkable progress in language understanding and generation, decoder language models continue to exhibit striking failures on numerical data. Numerical examples are typically serialized as text by concatenating feature names with their values (e.g., fat:3.4), while conventional task adaptation applies loss onl...

Weimin Lyu, Jing-Feng Zhang, Ling-Dao Sha et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Entity Matching using Decision Models

Entity matching is the task of deciding whether two records describe the same real-world entity. Current entity matching methods rely on pre-trained language models (PLMs) which require task specific fine-tuning or generative large language models (LLMs) which do not require task-specific fine-tuning but are slow and e...

Aaron Steiner, Ralph Peeters, Christian Bizer · 0 citations

Related blog posts

Google DeepMind Blog Nov 18, 2026

Genie 4 is now available

We are rolling out a new Project Genie experience to eligible Google AI Ultra subscribers, opening access to our new Genie 4 world model.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.