LLMs Can Leak Training Data But Do They Want To? Evaluating Memorization Propensity in LLMs

Abstract

Large language models can reproduce training data, but memorization evaluations mostly measure whether models can be made to do so, not whether they do so without adversarial prompting. We introduce PropMe, a framework that pairs a capability evaluation (prefix attacks) with non-adversarial propensity evaluation, and a propensity metric to relate the two. We check every generation against the model's full indexed training corpus with SimpleTrace, an open-source batch implementation of OLMoTrace span matching. We evaluate Comma, its continually pre-trained descendant DFM Decoder, and OLMo3 32B on Common Pile, Dynaword, and Dolma3, with four non-adversarial settings ranging from no prompt to sentences from unseen in-domain sources. Our results show that natural sentence prompts reproduce far less training text than prefix attacks. Generation with no prompt or a single common word, however, reproduces about as much as prefix attacks, or more, in three out of four model-corpus pairs; most of this text is duplicated licenses, code, and templates. In our setting, continual pre-training reduces long verbatim reproduction of the original corpus only modestly. More broadly, we argue that by testing both capability and propensity, we can attain a more comprehensive picture of a model's memorization behavior.