Can Open-Weight LLMs Predict Test Outcomes from Traces? A Cross-Subject Boundary Study on Tests4Py

Abstract

Determining whether a test execution should pass or fail remains a core obstacle to end-to-end test automation. Recent work has focused mainly on generating assertions from source context, leav- ing open how much pass/fail signal can be recovered directly from runtime behavior, especially under subject transfer. We present a cross-subject boundary study of this question on Python unit tests from the Tests4Py benchmark. Our pipeline collects execution traces via runtime instrumentation, compresses them through redundancy pruning and two minimization strategies (holistic filtering and key- point amplification), splits long traces into chunks whose partial predictions are aggregated per test, and fine-tunes open-weight large language models (LLMs) to classify tests from previously un- seen subjects as passing or failing. We evaluate CodeT5-small, Phi- 3.5-mini-instruct, and Llama-3.1-8B under varying context budgets and overlap settings. The best completed configuration, CodeT5- small with keypoint amplification and no overlap, reaches a macro- averaged failing-class F1 score of 𝐹 1fail = 0.365 across the three held-out subjects of our fixed protocol, compared with 𝐹 1fail = 0.025 for an empirical-distribution reference; since the released traces contain no exception events, this signal cannot stem from spotting recorded exceptions. Larger context windows improve the holistic filtering configurations, whereas overlap has mixed effects. Within this single-protocol setting, runtime traces contain a non- trivial cross-subject signal. However, performance remains too low and too unstable across subjects for stand-alone deployment. In this setting, progress appears to depend more on representation, aggregation, and robustness to subject shift than on model scale alone.