Sweat the Details: Exercise Breaks and AI Assistance in Code Summarization
TL;DR
A controlled human-subjects study examining the effects of physical exercise and AI-generated code summaries with varying correctness on developers’ summarization and bug detection performance concludes that AI assistance is generally useful for code summarization, but incorrect AI assistance substantially degrades bug detection performance.
Abstract
—Code summarization supports program comprehension and defect detection, and developers increasingly use both human-centered and AI-based interventions when performing this task. We present a controlled human-subjects study ( N = 47 ) examining the effects of physical exercise and AI-generated code summaries with varying correctness on developers’ summarization and bug detection performance. Participants summarized GitHub code under different intervention conditions, with outcomes evaluated along multiple dimensions, including accuracy, completeness, conciseness, readability, defect detection, and response time. We analyzed the data using mixed-effects models to account for repeated measures across participants and code artifacts. Surprisingly, under the exercise intervention studied, we did not observe consistent benefits for summarization quality. AI assistance is generally useful for code summarization, but incorrect AI assistance substantially degrades bug detection performance. These findings provide empirical evidence on the nuanced benefits and risks of human-and tool-based interventions in code summarization and bug detection.