A Language Model from 1913: Pretraining on Historical Text
TL;DR
This work constructs TypewriterCorpus, a 54B-token historical corpus with extensive temporal filtering, proposes lexically grounded instruction tuning that constrains all responses to vocabulary from historical source documents, and introduces History-Event, a benchmark of 2,344 events for evaluating both competence and cutoff adherence.
Abstract
While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text) in a data-constrained setting can produce a temporally grounded language model that still shows reasonable performance on language understanding. However, developing History LMs requires addressing challenges in data quality, preventing temporal leakage in post-training, and constructing temporally aligned evaluations. We address these challenges and pretrain TypewriterLM, a 7.24B-parameter model with a 1913 knowledge cutoff. We construct TypewriterCorpus, a 54B-token historical corpus with extensive temporal filtering, propose lexically grounded instruction tuning that constrains all responses to vocabulary from historical source documents, and introduce History-Event, a benchmark of 2,344 events for evaluating both competence and cutoff adherence. We release TypewriterLM and all associated resources to support future research on History LMs.
