Skip to content
Preprint

When Code Gets Cheap, Evaluation Gets Scarce: An Experience Report on AI-Assisted Agile Software Engineering

Oct 2026 · 0 citations · 49 references
Computer Science

Abstract

Coding assistants are widely reported to speed up code production; most accounts stop there. We report the end-to-end construction and deployment of a compliance-critical, human-in-the-loop exam-grading system, built by a single developer who wrote no code by hand, protected by a gated process from stakeholder persona to audited behavior. The initial system was built in three working days and could grade a real cohort. We draw one claim from this experience: AI changes software engineering less by making programming faster than by changing what is scarce. Writing code becomes abundant. Reliable evaluation-knowing whether a result is correct and meets the actual need-remains necessary at every iteration and becomes the binding constraint. Engineering judgment remains a second scarcity. Several defects passed the test suite: they sat at the boundaries the tests had mocked, the browser and the language-model API, or appeared only in real use. Deployment exposed assumptions that local tests had missed. Moreover, an AI assistant acting across many steps bypassed process checkpoints until we made them enforceable gates. All of this was observed in a triple union: the developer was also the end user and domain expert, making expert judgment immediately available and evaluation as cheap as it can be. This favors the iteration gains, which may shrink where evaluation requires additional people or expertise, but makes the case a hard test for evaluation scarcity: evaluation still constrained progress. We draw five lessons: run a gated, traceable construction process; specify what is knowable and discover the rest against a running prototype; validate in the real environment; gate the assistant, not only the work; record decisions with their assumptions and the process state the assistant forgets. A second project showed similar patterns; an independent single-expert case corroborates the shift.

View source

Similar papers

#software testing Review Open access Oct 2026

From Code Author to Code Reviewer: A Systematic Review of Practitioner Experience with Generative AI in Software Engineering

This review examined 58 empirical studies published from 2024 to early 2026 on GitHub Copilot, ChatGPT, Claude, and Gemini using PRISMA-guided selection from five digital libraries to address productivity, role shifts, code quality, adoption behaviour, and evidence gaps, and foundings were synthesised into the AI Integ...

B. Isong, Tshipuke Vahangwele, Nosipho Dladlu · 0 citations
Open access Oct 2026

A LOW-CODE APPLICATION DEVELOPMENT PLATFORM INTEGRATING AI-ASSISTED DEVELOPMENT

The platform is built on an improved RuoYi-Vue-Plus framework with an enhanced permission system, integrates the magic-api script engine, integrates the visual design engine and sandbox technology, and builds its AI-assisted development capability on ByteDance's TRAE.

Fan-Hao Ye, Shi-Wei Xu · 0 citations
Review Open access Sep 2026

Generative AI Assistants in Software Development: Productivity Gains, Code Quality, and Security Trade-offs

Generative artificial intelligence (AI) coding assistants promise to shorten software-development tasks, yet task speed alone cannot establish an improvement in engineering productivity. This critical narrative review asks whether assistants help developers complete work faster without increasing functional defects, ma...

Rikelmy Rodrigues Menezes · 2 citations
Book Open access Oct 2026

If You Build It, Will They Come? Lessons from 20 Years of Industrial Use of a Code Transformation DSL

An experience study of RcScript, a proprietary code-transformation DSL used in production at Raincode and Raincode Labs for more than 20 years, and of the long-lived transformation platform built with it to modernise PACBASE-generated COBOL, which shows a consistent pattern of features that improve local readability ar...

J. Fabry, V. Zaytsev · 0 citations
Review Nov 2026

Level Up! AI-Driven Challenges and Adaptations to Code Review Practices

AI can write code faster than humans can review, causing a bottleneck and necessitating changes to code review practices. This bottleneck is hotly debated, giving rise to two extreme and orthogonal imagined futures: one where humans continue to be firmly entrenched in the review process, and one where AI agents merge p...

Kaia Newman, Christian Bird, Jenna L. Butler et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.