Intelligent Optimization Simulation Framework for Test and Evaluation of Autonomous Systems
TL;DR
Results were that adequate uncertainty quantification can be achieved with as few as 10 repeated runs per simulation scenario, sensor realism has a significant effect on failure rate, distinct differences between the two autonomies failure modes were identified, and the authors' efficient optimization/search methods identify critical performance regions in a small fraction of the number of simulations required by a naïve Monte Carlo search.
Abstract
This paper describes ongoing research and development of an efficient optimization/search–based modeling and simulation framework for rapidly identifying low-performance scenarios in advanced autonomous systems. Ensuring predictable, safe behavior across complex, integrated systems remains a core operational test-and-evaluation challenge. Our goal is to balance rigorous validation with timely deployment. We are developing TEAAS (Test & Evaluation of Advanced Autonomous Systems), a scalable, faster-than-real-time framework designed to uncover critical failure scenarios efficiently. Key features include GPU-accelerated parallel simulation and learning, computational intelligence–based search of optimal parameters, uncertainty quantification for reproducibility, and real-time physics-accurate sensor models. We conducted simulation experiments to evaluate and demonstrate the framework performance for two black-box ground-vehicle autonomous systems. Key results were that adequate uncertainty quantification can be achieved with as few as 10 repeated runs per simulation scenario, sensor realism has a significant effect on failure rate, distinct differences between the two autonomies failure modes were identified, and our efficient optimization/search methods identify critical performance regions in a small fraction of the number of simulations required by a naïve Monte Carlo search.