Aug 2026· Machine Learning and Knowledge Extraction· Vol 8, pp. 252· 0 citations· 29 references
TL;DR
Student-history metadata can influence LLM-generated grading scores despite explicit instructions to ignore it, and future LLM-based grading systems should separate answer-based scoring from learner-context-based personalization and validate score invariance under controlled learner-context variations.
Abstract
Background: Large language models (LLMs) are increasingly explored for automated educational assessment, while future educational platforms may combine grading, feedback, learner analytics, and personalization. The objective of this study was to determine whether student-history metadata can influence the numerical score assigned to the same answer. Methods: This study introduces GradeDrift-LLM, a controlled framework for measuring student-history-induced score drift in LLM-based automated grading. We evaluated 1000 Computer Science answers from 100 students across six student-history conditions and eight open-weight LLMs. For each grading instance, the submitted answer, question, reference answer, rubric-related information, scoring scale, and grading instruction were kept constant; only the student-history condition varied. Results: Across 39,997 valid paired comparisons, 83.92% showed no drift, 9.40% showed upward drift, and 6.68% showed downward drift. Mean absolute drift was 0.2137 points, and the 95th percentile absolute drift was 1 point. Positive-history frames tended to increase scores, whereas negative-history frames tended to decrease them. Drift was model-dependent, not uniformly explained by approximate scale, and present in both technical and argumentative answers; rare extreme deviations reached 10 points. Conclusions: Student-history metadata can influence LLM-generated grading scores despite explicit instructions to ignore it. Future LLM-based grading systems should separate answer-based scoring from learner-context-based personalization and validate score invariance under controlled learner-context variations.
E EduFairBench provides a reproducible methodology for jointly analyzing predictive performance, robustness, uncertainty, and feedback quality, providing a comprehensive methodological framework for the rigorous evaluation of LLM-based educational assessment systems.
W. Villegas-Ch., Aracely Mera-Navarrete, Fernando Zúñiga-Tello et al.· Frontiers in Artificial Inte...· 0 citations
ConsensusGrade provides a diagnostic framework for interpreting automated scores relative to observed human variability; inside-envelope rates should not be interpreted as stand-alone measures of grading accuracy.
Cătălin Anghel, A. Anghel, Mihai Vlase et al.· Applied System Innovation· 0 citations
Agentic tutoring systems increasingly rely on large language models (LLMs) to generate real-time feedback, yet risky outputs, such as answer leakage, contradiction, and hallucination, can directly affect learning. Answer leakage is a central concern in this setting. It exposes problem solutions to students and can be verified through human annotation. This article presents a pedagogical style-conditioned audit protocol for automated evaluators that detect answer leakage in LLM-generated tutoring feedback before deployment. Using 1100 mathematics problems from the MATH dataset and 4400 feedback items generated under Baseline, Direct, Socratic, and Encouragement styles, we compare rule-based, embedding-based, natural language inference-based, and two LLM-as-a-Judge evaluators from different model families (Qwen2.5-32B and Gemma3-27B) and validate them against a 314-item human-annotated gold standard. The results show that pedagogical style changes how answer-leakage signals appear in feedback and how reliably automated evaluators detect them. Rule-based and LLM-Judge evaluators that perform well under Baseline, Direct, and Encouragement show consistent degradation under Socratic feedback. The primary LLM-Judge (Qwen2.5-32B) achieves overall F1 = 0.726 but drops to 0.427 on Socratic items, and a second LLM-Judge (Gemma3-27B) reproduces this degradation with Socratic F1 = 0.342. The four factorial prompt-sensitivity variants reach overall F1 up to 0.844, but their Socratic F1 remains at or below 0.390. These findings show that automated tutoring-feedback auditing must be style-specific, prompt-calibrated, and human-grounded before deployment.
Wen-Zhuo Song, Xuechen Wang, Shuang Liang et al.· IEEE Transactions on Learnin...· 0 citations
It is shown that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values, and effective mitigation must be validated for the intended model and task or domain.
A. Kapetanović, Kemal Altwlkany, Andro Merćep et al.· 0 citations
Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faster and cheaper than human evaluation. However, a judge may produce consistent scores without necessarily agreeing with human evaluators. In this work, we study this issue using two local open-weight LLM judges, LLaMA-3-8B and Qwen2.5-7B. We evaluate 300 responses generated by an instruction-tuned GPT-2 (124M) model for 100 questions covering five categories: factual knowledge, instruction following, mathematics, reasoning, and writing. Each response is scored by nine human annotators and is evaluated three times by each LLM judge using the same rubric. We compare the judge scores with the average human scores using Pearson correlation, Spearman correlation, mean absolute error (MAE), signed bias, and self-consistency. LLaMA-3-8B shows a Pearson correlation of 0.275 with human scores, while Qwen2.5-7B achieves 0.340. Their MAEs are 27.71 and 18.64, respectively. Despite this limited agreement, both judges show high self-consistency, with exact consistency rates of 97.3\% for LLaMA-3-8B and 92.3\% for Qwen2.5-7B. These results show that high self-consistency does not necessarily indicate high agreement with human judgments. Our findings highlight the need to evaluate both consistency and human alignment when using local LLMs as automatic judges.
Large language models are increasingly used for automated grading, but final-score agreement does not reveal whether credit is assigned to the rubric element affected by a response change. This study introduces CreditTrace-LLM, a controlled framework for auditing directional responsiveness, credit locality, paraphrase stability, and correspondence with expert evaluations in rubric-guided grading. The evaluation used 500 response families across ten questions, equally divided between technical and argumentative tasks. Each family included a baseline response (C0), a positive intervention adding rubric-relevant evidence (C+), a negative intervention removing or weakening such evidence (C−), and a placebo paraphrase (CP) constructed through lexical or syntactic reformulation with the intention of preserving rubric-relevant meaning. Eight local open-weight LLM graders produced 15,999 structurally valid Gold-point-level outputs from 16,000 expected evaluations. Targeted Gold-point scores changed in the expected direction in 75.7% of C+ comparisons and 50.2% of C− comparisons. Localized directional success was lower under C+ and C−, at 31.5% and 21.3%, respectively, indicating frequent non-target score changes. Under CP, total-score stability was 75.4%, while complete-profile stability was 73.6%. Direction concordance with the mean expert score change was 69.8% for C+, 47.4% for C−, and 18.2% for CP. The CP condition also showed substantial variability in expert scoring, particularly for argumentative responses, indicating that intended rubric-relevant meaning preservation did not guarantee score invariance. Final-score behavior alone is insufficient for validating rubric-guided LLM grading. CreditTrace-LLM therefore evaluates target responsiveness, credit locality, paraphrase stability, and traceable Gold-point-level outputs as complementary diagnostic dimensions rather than as predefined criteria for acceptable grading performance.
Cătălin Anghel, A. Anghel, M. Craciun et al.· Computers· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
