Agreement and Run-to-Run Consistency of an AI Marking System against Examination Board Marks: A Case Study in A-level

Abstract

Large language model (LLM) based marking systems are increasingly used to mark examinationstylework, but two questions must be answered before their marks can be relied upon: howclosely do they agree with examination board marks, and how much do their marks change whenthe same work is marked again? This paper addresses both questions for GradeDrive, an AImarking system, on A-level Mathematics. We compare GradeDrive’s marks with AQA’s markson n = 185 question parts (520 marks in total) from 26 student scripts across Papers 1, 2 and3 of AQA A-level Mathematics (7357), drawn from AQA’s own examiner and teacher trainingmaterial for the 2019, 2022 and 2023 series. These responses were chosen by AQA for trainingpurposes and are deliberately contentious. GradeDrive’s marks correlated strongly withAQA’s (r = 0.884; quadratic-weighted κ = 0.864, 95% bootstrap CI [0.806, 0.909]; ICC(2,1)= 0.865), with exact agreement on 70.3% of question parts and agreement within one mark on93.0%. GradeDrive was systematically more generous than AQA (mean +0.30 marks per part;t(184) = 5.99, p < 0.001), over-marking on 25.9% of parts and under-marking on only 3.8%.The generosity was concentrated on responses AQA judged weak: where AQA awarded zero,GradeDrive awarded at least one mark on 44% of parts, whereas where AQA awarded full marks,GradeDrive agreed on 97%. To measure repeatability, 18 scripts were marked three times each.Script totals were identical across all three runs for 7 of the 18 scripts and within one markfor 12; the mean within-script standard deviation was 0.63 marks (ICC = 0.994). Run-to-runvariation was therefore small relative to the systematic offset from AQA, which averaged aboutthree marks per script. We compare these results with an earlier GradeDrive study on GCSEPhysics and with published estimates of examiner-to-examiner reliability.