Evaluating the Effect of Generative AI-Assisted Debugging on Students’ Critical Thinking
TL;DR
The findings suggest that GenAI tools like LLAMA may be associated with different patterns of critical thinking during debugging, yet their association with individual skills seems limited in the absence of well-structured instructional support.
Abstract
The growing presence of generative AI (GenAI) has raised questions about its role in supporting or hindering critical thinking in programming education. This study examined the influence of GenAI tools, particularly LLAMA, on students’ critical thinking skills during structured debugging tasks in a Java-based CS1 course. In a quasi-experimental design, 132 students were divided into an experimental group that received GenAI-assisted debugging instruction and a control group that received traditional instructor-led instruction. Both groups completed structured debugging tasks, a closed-book post-evaluation, and a program writing task, while students in the experimental group completed a survey. Students who used LLAMA for debugging tended to link lower- and higher-level thinking skills through stronger conditional associations between understanding code, applying solutions, and extending them into creative solutions, while students who used manual debugging showed stronger connections between analyzing problems and evaluating solutions. Permutation tests confirmed significant differences in skill associations between groups. However, these network-level differences did not always translate into higher performance scores on individual skills, as the control group achieved higher scores in core skills with lower-level cognitive demands. Creativity was also one of the differences observed between the groups. In the LLAMA group, Creativity was connected to Application, with a weaker connection to Analysis, whereas in the control group, it was disconnected from other skills. However, in some cases, students’ solutions in the control group showed structural changes that went beyond what was taught. The findings suggest that GenAI tools like LLAMA may be associated with different patterns of critical thinking during debugging. Yet, their association with individual skills seems limited in the absence of well-structured instructional support. To properly harness GenAI in debugging education, educators must adopt pedagogical approaches that guide students toward reflection, independent reasoning, and balanced cognitive engagement.