Toward Automated Search for Verifier-Spending Agentic Workflows: A Vision, Preliminary Evidence, and the Obstacles in the Way

Abstract

Agentic software-development systems increasingly wrap language models in workflows —repair loops, best-of-N samplers, multi-agent pipelines. We argue for a specific lens on why such structure helps and on how to design it automatically. Our position is that the value of an agentic workflow is the quality it converts a verifier into: a workflow is the control structure that turns a pass/fail signal (a test suite, a type checker, CI) into an edit, and a single monolithic model call—which emits an answer once and stops—structurally cannot spend a verifier at all. The design objective worth optimising is therefore verifier utilisation, and the natural tool is automated search over workflows that maximises it. We give preliminary evidence that verifier utilisation is the load-bearing variable (on a hard code benchmark, an executed test-repair loop moves a cheap model +14 points over an identical monolith), and we report, honestly, the obstacles we hit trying to make an automated search discover such workflows: agentic verifier-spending (a model choosing to call a test tool) is unreliable; prompting does not fix it; reliability requires treating verifier-spending as a deterministic operator. Once it is, a simple evolutionary search already selects the verifier-spending operator over verifier-blind alternatives; discovering such operators from scratch, and scaling the search, remain open. We frame the resulting research agenda.