A Criteria-Governed Human-in-the-Loop Information System Architecture for Explainable AI-Assisted Learning and Preliminary Assessment

Abstract

This paper presents a criteria-governed, human-in-the-loop information system architecture integrating formative AI-assisted learning with preliminary assessment of spoken and written responses. It supports two modes: a retrieval-augmented assistant answering from course materials and an AI examiner scoring responses against a predefined rubric, criterion by criterion. The language model is confined to analysis: evidence extraction, aggregation, input-quality control, uncertainty estimation and escalation are performed by a controlled pipeline, and each result is stored as a structured decision object carrying criteria, evidence, provenance, uncertainty and teacher-verification status. Across 168 examination sessions and 1074 criterion-level decisions, every decision carried a verbatim evidence passage (100% coverage), the overall score matched the programmatic aggregation in all finalized sessions, 39.1% of responses were routed for secondary analysis and 5.4% of sessions were escalated. A survey of 52 students rated question clarity 4.81, criterion-level feedback 4.46, and allotted time 4.42, while concerns persisted about scoring reliability and recognition errors. In a preliminary blind calibration, the AI ranked 13 spoken answers like human raters (r = 0.75), but scored them 22 points lower. The contribution is the architecture and its feasibility in deployment; the study does not establish automated-scoring accuracy, and final decisions remain with the teacher.