SEBGen: A Self-Evolving Benchmark Generation Framework for Software Engineering Agents

Abstract

Static software engineering benchmarks such as SWE-bench suffer from three fundamental limitations: task sets are fixed and become saturated as model capabilities rapidly improve; expert construction is costly and slow; and difficulty levels cannot adapt to evolving model abilities. We present SEBGen, a self-evolving benchmark generation framework that synthesizes evaluation tasks from code repositories and continuously updates its generation strategies through an evolution loop. SEBGen introduces three key innovations: an explicit five-dimension exploration value evaluator that quantitatively scores generation strategies across quality, novelty, difficulty coverage, skill diversity, and evolution potential, replacing implicit LLM judgments; a hybrid validation pipeline combining three mechanical checks (schema, compile, functional), an LLM-based semantic assessment, and a repeated-judgment gate; and a self-evolving loop with archive-based experience accumulation, sigmoid-scaled parent selection, and LLM-driven strategy mutation. On a synthetic corpus, SEBGen achieves a 96 percent stage-level validation pass rate (E1; 48/50 stage observations) and preserves generation quality through the evolution loop while showing no measurable improvement over a budget-matched non-evolution baseline. On a real repository, it generates candidate benchmark tasks from all six tqdm commits that passed the schema and LLM-based screening stages, although the compile and functional stages could not be executed because these tasks carry no test patch. A judge-level comparison against SWE-bench Lite, partially corroborated by an independent judge from a different model family (Qwen), places generated real repository tasks below expert-constructed tasks (0.48 vs. 0.83 under deepseek-chat; 0.55 vs. 0.78 under qwen-plus, mean of two passes). An extension to two further repositories (requests, flask) achieves 0/4 acceptance across executed screening stages, indicating that the system does not yet transfer reliably beyond the original repository. These results provide proof-of-concept evidence of feasibility: the framework generates candidate tasks automatically, while executable verification of real-repository tasks, reliable self-improvement, and cross-repository transfer remain open problems.