Coverage-Free LLM-Based Fault Localization via Semantic Reasoning Over Historical Bug Reports

Abstract

Continuous Integration (CI) facilitates a continuous development flow by automating build and test processes and providing rapid feedback. Such feedback often reports build or test failures, which indicate faults in the system under test (SUT) and initiate the debugging process, typically starting with fault localisation. Recent studies have employed Large Language Models (LLMs) to improve fault localisation by exploiting program and failure semantics. However, most existing approaches rely primarily on information from the current failure instance. In practice, developers frequently repeat similar mistakes; consequently, recently reported related issues and bug reports often describe recurring failures and may provide additional cues for localising new faults. In this work, we propose REPORTFL, a novel LLM-based fault localisation approach that extends an existing coverage-based LLM localisation framework by replacing its reliance on coverage information with semantic reasoning over bug reports, retrieving and exploiting semantically related past issue reports alongside the current report. Our approach aims to examine the usefulness of lexical and semantic context from current and relevant historical issues as an additional reasoning signal. REPORTFL assumes a realistic coverage-free CI scenario, in which only the failing test name and bug reports are available. An experimental evaluation on 302 real faults from open-source projects shows that, compared to the coverage-based LLM technique it builds upon, REPORTFL achieves comparable localisation performance when using GPT-3.5 and outperforms it when using a more capable model, GPT-4.1-mini, despite not relying on coverage information. Controlled ablations, retrieval-window analyses, and a random-report-selection baseline further show that semantically relevant historical issue reports can provide useful reasoning context for LLM-based fault localisation.