Rethinking Automated Scoring in the Era of Artificial Intelligence: Emerging Directions for AI-Based Assessment

Abstract

The rapid development of generative artificial intelligence (AI), particularly large language models and multimodal models, is reshaping research on automated scoring in educational assessment. While previous research has often focused on the extent to which AI-generated scores agree with those assigned by human raters, agreement alone may not provide sufficient evidence for the quality or defensibility of an automated scoring system. This editorial argues that next-generation AI-based scoring research should move beyond model-level performance comparisons and examine the broader conditions under which AI-generated scores are produced, evaluated, and used. It first discusses key design dimensions of AI-based scoring systems, including prompt design, rubric design, example-based calibration, model selection, scoring strategies, and continuous performance monitoring. It then identifies emerging research directions involving evidence-grounded scoring, uncertainty and confidence calibration, process- and reasoning-aware evaluation, multimodal evidence, retrieval-augmented scoring, fairness, and human–AI orchestration. Particular attention is given to iterative multi-agent and deliberative scoring, in which AI systems can review, critique, and reconsider previous scoring decisions rather than independently producing a final score. The editorial concludes by proposing a broader research agenda in which automated scoring is evaluated not only in terms of agreement or accuracy, but also validity, reliability, fairness, uncertainty, transparency, and the conditions under which human oversight should be incorporated.