Could LLMs Predict Algorithm Performance in Automated Design of Metaheuristics?
This study investigates whether the current generation of frontier Large Language Models can predict the relative performance of metaheuristic optimization algorithms from source code alone, potentially reducing the computational cost of benchmarking during automated algorithm design. We task three models (GPT 5.4 Thinking, Gemini 3.1 Pro, and Claude Opus 4.6) with predicting improvement or regression between consecutive iterations of iteratively refined differential evolution, and with ranking independently designed algorithms on standard benchmarks. Initial results are promising (88% accuracy), but accuracy collapses on a second algorithm sequence (22–44%). A controlled comparison experiment provides evidence that the models' predictions correlate with code complexity metrics rather than with actual algorithmic performance. Self-reported confidence scores are poorly calibrated across all models tested. These findings indicate that while LLM-based performance prediction is not yet a reliable substitute for benchmarking, it shows potential as a complementary pre-screening tool in iterative algorithm design, particularly in settings where code modifications follow predictable patterns.