Don’t Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality
The first controlled longitudinal study that isolates the scaffolding’s contribution to agent quality over time is conducted, revealing that despite continuous development activity and growing codebase complexity of the scaffoldings, there is no statistically significant improvement in SWE-bench benchmark score across releases for a given fixed LLM version.