FitGen: Validation and Guided Repair of LLM Generated Executable Fitness Functions for Search Based Software Testing
TL;DR
FitGen is presented, a validation guided repair framework combining structured prompt design with output, syntax, and semantic validation to automatically diagnose and repair executable fitness functions.
Abstract
Search-Based Software Testing relies on fitness functions to guide search algorithms toward target branches. Traditionally, computing fitness for branches involves a program analysis pipeline, with instrumentation that is language and type dependent. Large Language Models (LLMs) provide a promising mechanism for automatically generating branch aware executable fitness functions. However, directly using LLM generated outputs is unreliable, leading to incomplete outputs, compilation errors, or incorrect optimization objectives. This paper presents FitGen, a validation guided repair framework combining structured prompt design with output, syntax, and semantic validation to automatically diagnose and repair executable fitness functions. We evaluate FitGen on 11 benchmark programs containing 51 target branches using fitness functions generated by six state of the art LLMs. Our results show FitGen improves branch validation performance across all LLMs by up to 40.91% using the repaired fitness functions, and search based optimization achieves up to 93.59% branch validation accuracy.
