Self-Regulation, Scaffolding, and the Illusion of Improvement: A Quasi-Experimental Evaluation of an LLM-Based Programming Help Tool
LLM-based programming help tools integrated into learning management systems offer new possibilities for supporting students in large programming courses. Yet existing research rarely accounts for the self-regulatory differences that shape who chooses to use these tools or for the ways that technological scaffolds interact with learners’ motivation and help-seeking behaviors. This study addresses that gap through a quasi-experimental design that intentionally delayed the introduction of an LMS-integrated LLM tool, CodeHelp, until after early-semester assessments and substantial measures of student effort had already been collected. Using data from 589 students in a second-level programming course, we find that students who chose to use the tool were already more engaged before it became available: they attended more classes, spent more time in labs, and earned higher scores on the first exam. When these differences were accounted for using logistic regression and propensity score matching, the apparent performance benefit associated with CodeHelp disappeared. These findings suggest that engagement with LLM-based tools reflects underlying self-regulatory behaviors and that the tool functions as a form of technological scaffolding primarily activated by already-engaged learners. Methodologically, the study demonstrates how delaying tool introduction and applying causal inference methods can produce more credible estimates of impact in real classrooms. Pedagogically, it reveals that LLM-based programming support may amplify existing disparities in self-regulation rather than compensate for disengagement. As enthusiasm for generative AI in computing education grows, this study highlights the need for transparent, theory-informed evaluation practices that distinguish genuine learning gains from pre-existing differences in student behavior.