Whose Specification Did You Test? The Hidden Interpretation Bias in LLM-Generated Test Suites
TL;DR
This work examines three test-generation strategies that vary in coverage and emphasis on ambiguous behaviours and discusses the implications for autonomous software engineering pipelines in which a single model generates both code and tests from the same specification.
Abstract
LLMs are increasingly used to generate both software implementations and the test suites used to evaluate them. This raises a fundamental question: when an API specification leaves behaviour under-specified, do LLM-generated tests act as neutral observers of correctness, or do they encode interpretation choices that favour implementations sharing those choices? We investigate this question through a controlled experiment in which identical LLM-generated test suites are applied to independently developed human-coded and LLM-coded REST API implementations. We examine three test-generation strategies that vary in coverage and emphasis on ambiguous behaviours. The results provide statistically significant evidence of interpretation-coupled testing: test outcomes depend partly on agreement between the test suite's implicit interpretation and the implementation's interpretation of the specification. We discuss the implications for autonomous software engineering pipelines in which a single model generates both code and tests from the same specification, and identify practical directions for interpretation-aware test design.
