This paper presents a search-based testing framework for LLM-based applications that incorporates failure diversity as an explicit optimization objective and maintains an archive of generated tests and rewards distance from that archive, while a repopulation operator periodically replaces non-failing tests in the population to sustain exploration.
Abstract
Large Language Model (LLM)-based applications are increasingly deployed across domains including customer service, education, and mobility. These systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging. In this paper, we present a search-based testing framework for LLM-based applications that incorporates failure diversity as an explicit optimization objective. Building on a discretization along stylistic, content-related, and perturbation dimensions, our framework maintains an archive of generated tests and rewards distance from that archive, while a repopulation operator periodically replaces non-failing tests in the population to sustain exploration. The repopulation operator is parameterized by its sampling strategy: replacement candidates are drawn either uniformly or by greedy distance maximization, two settings we compare empirically. We evaluate both against several baselines across three case studies---LLM safety, in-car navigation, and in-vehicle function control---covering five systems under test, eight LLMs, and 18 distinct test configurations, with over one million executed tests. Our results show that all guided variants detect substantially more failures than random and combinatorial search in nearly all configurations. Among them, diversified search detects fewer failures, but covers a broader range of failure types in all case studies, with greedy repopulation offering the best tradeoff.
The first compute-normalised comparison of five TTS families across five open-ended generation benchmarks spanning medicine, law, finance, general chat, and creative writing is conducted - grounded in a unified framework that decomposes the effectiveness of each method's token budget into exploration and exploitation.
Davide Romano, Kanak Raj, Jerrod Parker et al.· 0 citations
This paper formalizes test suite generation as a Markov Decision Process (MDP) and demonstrates that its objective exhibits monotone submodularity, which enables an effective relaxation of this NP-hard global optimization into a tractable step-wise greedy procedure.
Guo-Qing Wang, Cheng-Ran Yang, Xiao-Xuan Zhou et al.· Proceedings of the ACM on So...· 0 citations
This work extends Compute-Aligned Training to this setting through an abstraction of policy-guided search, deriving tractable, trace-supported losses and introduces a search-agnostic uniform-allocation (UA) loss that accounts for the budget without specifying the specific search.
SEDIMA is introduced, a persistent hierarchical insight memory for evolutionary search agents that distills raw traces into natural-language insights, clusters them by semantic similarity using attention-weighted centroids, and retrieves relevant guidance to condition future mutations, accumulating transferable knowled...
Amirhossein Abaskohi, Mahdi Mostajabdaveh, Zi-Rui Zhou· Proceedings of the Second Wo...· 0 citations
This survey systematizes recent progress in tree-search-based reasoning, viewing inference as instance-specific optimization rather than decoding, and introduces a Unified Design Space spanning search topology, evaluation signals, and control dynamics to unify a fragmented literature.
Jia-Qi Wei, Xiang Zhang, Yue-Jin Yang et al.· 0 citations
Experiments across diverse reasoning tasks, including MATH500, HumanEval, and GPQA, show that APS consistently outperforms power sampling with a fixed sharpening exponent, without additional training.
Bing-Nan Xiao, Chen-Hao Yang, Bing-Cong Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.