Analyzing Generative AI-Based Automated Scoring of Problem-Solving Competency: A Focus on Discrepancies with Human Scoring

Abstract

This study examined the potential and limitations of generative AI-based automated scoring of students’ problem-solving competency by focusing on discrepancies between human and AI scoring. Thirty social issues inquiry reports written by 12th-grade students were scored using the same rubric by five expert raters and the Gemini 3 Pro model. Agreement among human raters was first examined, followed by quantitative and qualitative comparisons between human and AI scores. The five human raters showed very high agreement (ICC = 0.966). Although human and AI scores were relatively highly correlated (r = 0.796), their agreement was moderate (QWK = 0.554; ICC = 0.556). Among 300 scoring instances, 81 discrepancies were identified. Qualitative content analysis revealed seven major types of discrepancies: (1) lenient scoring under ambiguous rubric criteria, (2) assigning credit based on partial cues, (3) sensitivity to evaluative wording, (4) overreliance on scoring criteria, (5) limitations in judging logical coherence, (6) misinterpretation of document structure and boundaries, and (7) other discrepancies. These findings demonstrate both the potential and limitations of generative AI-based scoring for performance tasks requiring professional judgment and suggest the need for complementary use of human and AI scoring.