Using LLMs to Detect Growth in Computational Thinking in Introductory Physics
TL;DR
This study investigated the use of Large Language Models to analyze students' explanations of computational physics problems on a pre- and post- semester survey and found significant growth in Data Practices and Computational Problem-Solving Practices.
Abstract
As computation becomes more central to physics education, creating scalable methods to assess authentic computational thinking (CT) in students remains a critical challenge. While student-written responses capture nuanced reasoning, they are difficult to evaluate at scale. In this study, we investigated the use of Large Language Models (LLMs) to analyze students'written explanations of computational physics problems on a pre- and post- semester survey. By first establishing a human-coded baseline, grounded in CT literature, we identified significant growth in Data Practices and Computational Problem-Solving Practices. When given the same responses, an LLM successfully mirrored the human evaluations and scaled up the detection of these key trends across a large dataset. Notably, both human raters and the LLM struggled to reliably evaluate more complex constructs such as Systems Thinking. Overall, this study demonstrates that LLMs offer a viable method to scale the evaluation of students'CT in large-enrollment physics courses