Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses
TL;DR
The study shows that student engagement with LLMs varied across courses and increased over the term, and although students expressed high satisfaction and reported saving considerable time, their perception of deep learning benefits declined, and many noted a tendency toward over-reliance.
Abstract
This paper presents a multi-source classroom study conducted during a 10-week quarter in data science courses at Drexel University. We first investigate the behaviors of student engagement with large language models (LLMs) using four surveys across three data science courses. Second, we evaluate the construct validity of multiple-choice questions (MCQs) generated by an LLM for in-lecture retrieval practice. Based on 378 authored questions (311 deployed, producing 7{,}888 student responses), we analyze whether the difficulty ratings assigned by an LLM match empirical item difficulty. Our study shows that student engagement with LLMs varied across courses and increased over the term. Although students expressed high satisfaction and reported saving considerable time, their perception of deep learning benefits declined, and many noted a tendency toward over-reliance. Regarding the difficulty ratings of LLM-generated MCQs, the Easy, Medium, and Hard labels correlated closely with its assigned Bloom's Taxonomy levels (Spearman $\rho=0.90$), reflecting an artifact of co-generation. However, neither metric predicted empirical item difficulty (difficulty label $\rho=0.06$; Bloom level $\rho=0.02$). The ratings reflect the structural formatting of a question rather than its underlying difficulty.