A Comparison of the Relative Effectiveness of Student SelfAssessment/Remediation vs. Using a Self-Assessment Chatbot on Raising Student Performance

Abstract

In previous research, we have shown that teaching students to self-assess and remediate their own knowledge (gaps) or giving them self-assessment chatbots that take student self-assessments as inputs and use those inputs in answering students’ questions both lead to large gains in student performance. The present study compares the relative effectiveness of each approach to see whether adding a chatbot to aid in remediation provides any benefit to having students selfremediate without the aid of technology. Forty fourth- and fifth-grade students studied a geometry unit on angle relationships and completed identical instruction followed by a Cognitive Structure Analysis (CSA)-based self-assessment that identified strengths and deficiencies in four knowledge categories: facts, strategies, procedures, and rationales. Participants were randomly assigned to one of two remediation conditions. The control group used its self-assessment to guide independent review of the instructional materials, whereas the experimental group submitted the same self-assessment to an LLM-powered chatbot that generated personalized explanations and guidance targeted to each learner's reported knowledge gaps. Learning was measured using parallel pretests and posttests. Both groups demonstrated statistically significant gains from pretest to posttest (both p < .0001). However, students using the self-assessmentinformed chatbot improved by 32.6 percentage points compared with 21.2 percentage points for students performing self-assessment without chatbot support, a statistically significant difference of 11.4 percentage points (t(38) = 4.02, p = .0003). These findings suggest that combining learner-generated knowledge models with LLM-based conversational tutoring produces substantially greater learning gains than self-directed remediation alone.