A Quasi-experimental Study of Conversational AI-supported Engagement in Large CS Courses

Abstract

Lucrative career prospects and creative opportunities often attract students to enroll in computer science courses, resulting in large class sizes. A common challenge in large classrooms is the lack of engagement between students and both the instructor and the learning material. Recent advances in Large Language Models (LLMs) present an opportunity to leverage Conversational Artificial Intelligence (CAI) to address this challenge. To examine the potential of CAI to support engagement with learning material, we designed an in-class activity in a large Software Engineering course in which students interacted with a CAI tool. Using a quasi-experimental, within-subject study design in a real classroom at a large public United States University. We compared student engagement during an activity that involved the use of CAI for discussion with an activity that did not (peer discussion). For our analysis, we collected data on students’ interactions with the CAI tool and their self-reported responses on multiple engagement scales. Results show higher student engagement with the learning material in the activity involving CAI discussion as opposed to without (peer discussion). Higher self-reported learning was also observed in the activity with CAI as compared to without. Students also reported significantly more usefulness for learning in the CAI condition. Overall, our results indicate that CAI (ChatGPT) has the potential to support engagement with learning content during in-class activities in large class sizes.