Passing Parity Is Not Preserving Behaviour: Oracle Escape in Function-Level LLM Code Migration

Abstract

Parity testing validates software modernization: the legacy system supplies every expected output, so the oracle is sound. One decision is delegated, increasingly to a language model and often the very model that translated the code: which inputs to try. How much divergence such suites let through is unmeasured, and whether a model validating its own translation misses more is unknown; we pre-registered that self-validation hypothesis. We measure both in Python-to-Java function migration with five models, each acting as both translator and validator, with expected values always from executing the legacy source and divergence established by differential execution on a large type-directed input set together with the generated suites. Of the parity suites that passed, 40.8% were on divergent translations, and 7.7% on translations that diverge on an input a reviewer would call unremarkable. We could not detect a self-validation penalty. Validator strength and translator subtlety dominate instead: misses are worst when a weak validator checks a strong translator's code, the configuration a cost-conscious pipeline prefers. A passing parity suite is evidence about where testing has looked, not a verdict of behavioural equivalence, and should be reported with its escape rate attached.