Skip to content
Book Open access

From Correctness to Expert-Refined Misconception Signals with Human-in-the-Loop Validation in CS1 Programming Education

Sep 2026 · UK & Ireland Computing Education Research Conference · pp. 1-7 · 0 citations · 16 references
Computer Science

TL;DR

The study shows that richer learner representations can move CS1 knowledge tracing beyond predicting outcomes alone towards explaining the nature of student struggle in ways instructors can interpret and use.

Abstract

Knowledge tracing in introductory programming is commonly framed as predicting whether a student will eventually solve the next problem. While useful, this approach reduces rich programming activity to correctness labels and offers limited support for instructional interpretation. This study examines whether learner modelling in CS1 can be improved by moving from correctness histories to expert-refined misconception signals derived from programming traces. Using CodeWorkout trace data, we compared three models on a held-out later-course split: a correctness-only model, a process-aware model using compilation, execution, and revision features, and a misconception-signal model that added expert-refined recurrent code-pattern signals. Because eventual correctness was highly imbalanced, we evaluated performance using macro F1, AUC, and Brier score rather than relying on accuracy alone. The process-aware model achieved the strongest macro F1, while the misconception-signal model achieved the best AUC and Brier score, indicating stronger discrimination and calibration than correctness-only histories. To examine pedagogical value, we conducted a human-in-the-loop evaluation. Experts judged 90.0% of candidate signals plausible and 90.0% actionable for teaching, and misconception-aware summaries were preferred in 90.0% of paired instructor evaluations. These summaries were also rated higher for credibility, usefulness, clarity, and actionability. The study shows that richer learner representations can move CS1 knowledge tracing beyond predicting outcomes alone towards explaining the nature of student struggle in ways instructors can interpret and use.

Read PDF

Similar papers

Preprint Oct 2026

Bias and Reliability in Learning-Style Inference from Student--AI Dialogues: An Evaluation of GPT-4o and Gemini 2.0 Flash

Large Language Models (LLMs) are increasingly integrated into educational technologies for personalized support, creating opportunities for real-time learner modeling from dialogue. This study evaluates whether GPT-4o and Gemini 2.0 Flash can infer three Felder-Silverman Learning Style Model dimensions (Active-Reflecti...

Samaneh Zamanifard, Sajad Goudarzi, A. Robb · 0 citations
#artificial intelligence Preprint Oct 2026

TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in t...

Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad et al. · 0 citations
#artificial intelligence Preprint Oct 2026

SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair

Large language models are increasingly used in computer science education. However, the teaching abilities of even the most powerful LLMs are limited: they tend to reveal the solution prematurely and may lose focus on the learner's current debugging objective. To address this, we introduce SocraticRepair, a planning-gu...

Mykhailo Kopytskyi, Jian-Hua Yang, M. Seyedebrahimi · 0 citations
Open access Oct 2026

Automated Misconception Diagnosis & Pedagogical Remediation: Using Generative AI to Identify and Remedy Structural Bugs in Introductory Code

Novice programmers in introductory computer science (CS1) courses frequently experience cognitive overload, helplessness, and attrition due to structural code bugs. Unlike superficial syntax errors caught by standard compilers, structural bugs stem from fundamental misalignments between a student's mental model and the...

Bhanupriya Sharma · 0 citations
#natural language process... Preprint Oct 2026

Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

Large language models (LLMs) promise to automate one of higher education's most resource-intensive activities: assessing student writing. Existing evaluations are typically limited to a single model, a single prompt and a single institution. Using 761 undergraduate psychology assignments from three UK universities, we...

Giulio Corsi, Maryam Abo-Tabik, Yael Benn et al. · 0 citations
Book Open access Oct 2026

Correctness, Convergence, and AI-Generated Code Detection: A Longitudinal Study of Student and Large Language Model Code in Introductory Programming

Large language models can generate plausible solutions to programming assignments, making it tempting to detect their use by matching student code against a reference bank of generated solutions. Yet similar code can also arise when an assignment admits only a few natural implementations, which leaves open what a match...

Runlong Ye, Jing Fan, Angela M. Zavaleta Bernuy et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.