Efficient and Effective Test Generation and Type Inference for Python Applications

Abstract

Python has become one of the most widely used programming languages, valued for its flexibility and ease of use. However, its dynamic nature provides only limited static guarantees, making programs more difficult to write reliably: errors are more likely to remain undetected until runtime. Tests and type checking can mitigate these issues by detecting bugs—including runtime errors—before release, but creating tests and the type annotations required for checking demands substantial developer effort, discouraging their widespread use. Approaches to automating test generation and type inference are hindered not only by Python’s dynamic features but also by the substantial performance limitations of CPython, the standard Python interpreter, which make dynamic analysis costly in practice. In this dissertation, I explore how low-overhead dynamic analysis can be used to improve the reliability of Python software. First, I introduce SlipCover, a code coverage measurement tool that significantly reduces runtime overhead through efficient bytecode instrumentation and dynamic de-instrumentation. SlipCover enables practical use of coverage tracking in iterative workflows by achieving near-zero overhead, supporting a range of software engineering applications. Building on SlipCover, I next present CoverUp, a regression test generation tool that uses coverage feedback to efficiently guide a large language model in generating unit tests that significantly outperform previous methods in coverage. Finally, I propose RightTyper, an efficient and effective type annotation system. RightTyper combines types observed through low-overhead, statistically guided runtime sampling with static analysis and name resolution to produce type annotations comparable in quality to current LLM-based approaches and substantially better than those of all other evaluated tools—while remaining grounded in observed program behavior rather than probabilistic inference. Collectively, these contributions advance the state of the art by showing how low-overhead dynamic analysis helps overcome significant challenges in writing and maintaining reliable Python code.