CRITERION-GUIDED AI ARCHITECTURE FOR LEARNING SUPPORT AND PRELIMINARY STUDENT ASSESSMENT

Abstract

This paper presents a pilot evaluation of a criterion-guided human-in-the-loop architecture for an intelligent educational system that supports learning and preliminary assessment of students’ written and oral responses. Instead of assigning a final score directly, the system combines relevant context, formalized criteria, interpretable response features, supporting evidence, uncertainty estimates, and instructor verification. The architecture separates probabilistic components, including retrieval, automatic speech recognition, natural language processing, and a language model, from deterministic assessment and governance mechanisms such as rubric weights, aggregation rules, thresholds, audit logging, and referral rules. Two modes were implemented: a retrieval-augmented generation assistant for formative learning support and an AI examiner for preliminary criterion-based assessment. In the assistant pilot, 336 of 347 queries were processed without recorded technical errors, giving a completion rate of 96.8%; source citations were included in 89.0% of responses, and median response time was 980 ms. Student perceptions of the examiner were evaluated using 52 questionnaires on a five-point Likert scale. Mean ratings were highest for question clarity at 4.81, usefulness of criterion-level feedback at 4.46, and sufficiency of response time at 4.42. Students also reported notable uncertainty about the grade, with a mean rating of 3.79, and concern about speech-recognition errors, with a mean rating of 3.31. Among 45 oral responses, 12 (26.7%) lacked usable word-level timestamps and were referred for review. A worked case demonstrated traceability from response features and evidence to criterion scores, uncertainty handling, aggregation, and instructor verification. A preliminary comparison of 23 paired AI and instructor-released scores was treated as diagnostic because of a severe ceiling effect. The results support technical feasibility and procedural acceptability, but not yet the validity or reliability of automated assessment.