Sounding Like Which AI? Machine-Style Overlap of arXiv Abstracts with Language Models of Different Vintages
Abstract
We ask not how much text was produced with LLM help but how much writing, by any author, falls on the LLM side of a stylistic boundary, and which LLMs define that side. We define Machine-Style Overlap (MSO): the share of documents that a calibrated classifier on style features that ignore content-word identity judges at least as likely to be LLM output as pre-ChatGPT human writing. In 131,448 arXiv abstracts (2015–2026), MSO was 9.2% before ChatGPT and by June 2025 stood 25.4 percentage points (pp) above its pre-ChatGPT trend, far above placebo gaps. Because recent LLMs learn from recent writing, we compare generators by vintage. Against generators whose pretraining data predate ChatGPT, the gap is smaller (7.5–13.1 pp by post-trend fit; 3.5–7.6 pp for two models run from titles). Single-generator overlap mostly rose then fell, peaking later for newer generators even at equal text length (exploratory rank correlation 0.79, n = 18; 0.66 without three added post hoc). A fixed reference set, like a fixed marker-word list (delve), can show the shift receding while overlap with newer models still rises. In US school essays, English learners' essays fall in the region less often, largely because of essay scores. AI use: the study was carried out by an AI system (Claude Opus 5.5, Anthropic) under the author's direction and is disclosed in full in the paper's Acknowledgements. The author read and approved this version. Code, data and results: https://doi.org/10.5281/zenodo.23343987
