MASSIV: A Hybrid RAG for Automated Defeater Extraction in Autonomous Driving Safety Cases

Abstract

Safety cases for Automated Driving Systems (ADS) rely on structured arguments supported by evidence, yet they are vulnerable to hidden assumptions and unknown counter-evidence (“defeaters”). The manual review of growing volumes of natural-language incident reports exceeds human capacity. Here, we present the core tool of the Monitoring Assurance Indicator Validation framework "MASSIV": a traceable, closed-corpus hybrid Retrieval-Augmented Generation (RAG) pipeline. The tool combines sparse and dense-retrieval to identify relevant excerpts from accident investigation reports. It then prompts a large language model to generate defeaters grounded in the retrieved evidence. Evaluation of the Open Autonomy Safety Case Framework (OASCF) produced 167 defeaters, assessed by three domain experts. Aggregated results of 0.64 Precision@5, indicating the top-5 retrieval relevance and approximately 33% strong defeaters. The tool reduces manual corpus review while maintaining traceability essential for safety assurance. Our tool and dataset are released as an open artifact.