A Multi-Study Evaluation into Generative Artificial Intelligence for Test-Driven Development
TL;DR
GAI4-TDD makes failing tests pass with a success rate of about 90% on the first attempt; improves external/internal quality of software and students’ productivity; and professionals are generally-positive, although some barriers to GAI4-TDD adoption emerged.
Abstract
Test-Driven Development (TDD) is an agile practice that alternates writing failing (unit) tests (Red phase), implementing production code that makes failing tests pass (Green phase), and improving the code through the Refactor phase. TDD promises better external and internal quality of developed software and developers’ productivity, yet it remains difficult to practice. In prior work, some of the authors of this paper proposed integrating Large Language Models (LLMs) into the workflow of TDD, specifically during the Green phase by generating production code that makes failing (unit) tests pass. A supporting tool named GAI4-TDD (Generative Artificial Intelligence for Test-Driven Development) was also proposed to operationalize the integration of LLMs into the workflow of TDD for embedded systems development. In this paper, we validate this integration through a multi-study evaluation that is not limited to embedded systems development and comprises: (i) a laboratory study; (ii) a controlled experiment with graduate students in Computer Science/Engineering; and (iii) three industry-based qualitative studies. The results show that GAI4-TDD (i) makes failing tests pass with a success rate of about 90% on the first attempt; (ii) improves external/internal quality of software and students’ productivity; and (iii) professionals are generally-positive, although some barriers to GAI4-TDD adoption emerged.