Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling

Abstract

Subword tokenization introduces a computational layer in language models, where many distinct token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-token loss across evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input.