Testing AI-Based Systems: A Systematic Mapping of Challenges and Approaches

Abstract

Artificial intelligence (AI) components are increasingly embedded in software, challenging established assumptions of software testing practice. This article presents a systematic mapping review of testing approaches for AI-based systems, grounded in ISO/IEC 29119 and ISO/IEC TR 29119-11. The contribution is a structured, traceable mapping from challenges to approaches at the level of AI-based systems. From 1212 identified records, 26 studies were included in the final mapping, complemented by seven ISO/IEC and ISO/IEC/IEEE normative documents that were analysed for normative guidance rather than as empirical evidence. The review identifies five recurring challenge categories—weak specifications and the test oracle problem, data dependence and test inputs, non-functional and socio-technical properties, complex and dynamic environments, and continuous evolution and adaptation. It synthesises five corresponding approach categories: metric-based and comparative testing, scenario-based testing and simulation, architecture-driven testing, continuous validation and post-deployment testing, and standards-aligned and process-oriented testing. Across the mapped studies, classical testing concepts are reinterpreted and extended rather than replaced. However, evidence is unevenly distributed across challenge–approach combinations: architecture-driven testing is represented by two primary studies, while socio-technical properties are addressed only by two secondary studies in the standards-aligned and process-oriented category. Seven of the twenty-five challenge–approach combinations remain empty.