PWCC: Probability-Weighted Combinatorial Coverage for Risk-Informed Scenario Testing of Autonomous Systems
Abstract
Scenario-based testing of autonomous systems faces a fundamental unsolved problem: knowing when to stop. Existing coverage metrics measure either structural diversity or combinatorial breadth, but neither tells you how much of the system’s expected real-world operation remains untested. A test suite can score well on every existing metric while systematically missing the conditions the system will actually encounter. We introduce Probability-Weighted Combinatorial Coverage (PWCC), a metric that simultaneously measures coverage of the Op- erational Design Domain (ODD) and test adequacy. PWCC learns a probabilistic model of expected operating conditions from oper- ational profiling data, uses it to assign probability mass to 𝑡-way combinations of scenario features, and declares testing adequate when the uncovered mass falls below a risk-calibrated threshold. This gives practitioners something no existing metric provides: a principled, quantitative answer to “have we tested enough?” Evaluated on four autonomous driving datasets against seven baselines, PWCC demonstrates three properties. First, it is the strongest predictor of fault detection where the operational dis- tribution carries signal, reaching Spearman 𝑟𝑠 = +0.981 on Deep- Collision — 0.167 above the best combinatorial baseline at a lower interaction order. Second, it suppresses false sufficiency: on Deep- Scenario, PWCC achieves FSR = UFC = 0 across all splits versus CC’s mean UFC ≥ 95.8%, false sufficiency eliminated entirely, and, judged at its own stopping point, surfaces 99.6% of failure-triggering com- binations where combinatorial saturation stops after 6.4% of the campaign; on failure-rich datasets UFC drops by up to 4.9×. Third, it is stable: the score varies by at most 0.006 across a ten-fold increase in Monte Carlo sample size and grows monotonically with suite size. These results show that operational probability weighting is the decisive factor in avoiding dangerous false sufficiency verdicts; and that PWCC transforms coverage from a descriptive statistic into a risk-informed stopping decision.