From Pareto to Neural: A Mathematical Survey of Multi-Objective Optimization Algorithms—With Applications to Software Testing

Abstract

Multi-objective optimization provides the mathematical foundation for reasoning about trade-offs in complex decision problems, from engineering design to resource allocation. Software testing exemplifies such problems: practitioners must simultaneously optimize for fault detection capability, code coverage, execution cost, and test suite diversity—objectives that are fundamentally incommensurable. Since the early 2000s, multi-objective evolutionary algorithms (MOEAs) such as NSGA-II, MOEA/D, and their many-objective extensions (MOSA; DynaMOSA) have served as the dominant mathematical framework for navigating these trade-offs through Pareto-front approximation with hand-crafted fitness functions. However, the recent emergence of reinforcement learning (RL) and large language models (LLMs) is shifting the optimization paradigm from numerical Pareto-front approximation toward neural, semantically aware decision making over learned representations. This paper presents a systematic mapping study of multi-objective optimization algorithms, tracing their evolution from classical Pareto-based methods toward AI-driven and hybrid approaches, with software testing as the primary application domain. We survey 120+ papers published from 2000 to 2025 and propose a novel five-level taxonomy (L1–L5) that classifies optimization approaches along the intelligence spectrum: classical MOEAs, ML-guided MOEAs, RL-driven optimization, LLM-driven optimization, and hybrid neuro-evolutionary systems. For each level, we analyze the mathematical problem formulations (Pareto optimality conditions, Markov decision processes, and neural loss landscapes), objective function design, algorithmic convergence properties, and computational complexity. We further conduct a cross-cutting mathematical analysis comparing these paradigms along dimensions of convergence, diversity, scalability, and interpretability. Our survey identifies critical open mathematical challenges: the lack of formal convergence guarantees for LLM-driven optimization, the under-exploration of many-objective (m≥4) formulations in AI-driven testing, the sample complexity of reinforcement learning for combinatorial test optimization, and the absence of standardized benchmarks with known Pareto-optimal frontiers. We conclude by outlining a research roadmap for the next generation of multi-objective optimization systems that combine the complementary mathematical strengths of neural function approximation and evolutionary diversity preservation.