Correct According to Whom? Auditing What LLM-Generated Tests Actually Check
TL;DR
It is found that LLM-based approaches can generate incorrect, overspecified, or incomplete test suites in all configurations, and that the configuration has less of an effect on the generated tests than expected.
Abstract
Unit testing is an important part of the software development process. But unit testing can be time-consuming and tedious, and so many people have turned to LLM-based approaches to automate the process. The problem is, a passing LLM-generated unit test suite provides evidence only that the code is correct with respect to the LLM's understanding, not the developer's. Our concern is that LLM-based approaches might come up with their own definition of "correct" behavior, separate from what the developer intended. Therefore, in this work, we investigate if LLM-generated tests verify the scenarios developers actually expect. Specifically, we propose a benchmark, ScenarioBench, of expert-curated test scenarios for a subset of 30 methods from the Defects4J dataset. We generate tests for these 30 Defects4J methods using five different state-of-the-art LLM-based techniques, as well as Claude Code, simulating different common configurations in software development: when a natural language specification is available, when it is not, when the code is correct, and when the code is buggy. We systematically compare the generated tests to the expert-curated test scenarios, and we find that LLM-based approaches can generate incorrect, overspecified, or incomplete test suites in all configurations, and that the configuration has less of an effect on the generated tests than expected. Our findings demonstrate how a passing LLM-generated test suite can provide developers with a false sense of security that their software, either manually written or AI-generated, is correct.
