Comparing the Marking Accuracy of an AI Marking System with Official Examination Board Marks and Experienced Teacher Judgement A Case Study in GCSE Physics

Abstract

Automated marking systems built on large language models are increasingly proposed as a way toreduce teacher workload and to provide fast, formative feedback on examination-style questions.Before such systems can be trusted for either formative or summative use, however, their markingaccuracy needs to be quantified against an appropriate reference standard, and the choice ofreference standard matters: examination board (definitive) marks are themselves produced byfallible human examiners operating under time pressure, so agreement with the board is a proxyfor accuracy rather than accuracy itself. This paper reports a case study comparing the marksawarded by an AI marking system, GradeDrive, against (i) official AQA GCSE Physics markscheme outcomes and (ii) the average mark independently awarded by a panel of experiencedphysics teachers, on a sample of n = 71 previously-marked exam responses drawn from AQA’sown examiner-training material (2019, 2022 and 2023 series) and deliberately comprising edgecaseitems on which experienced examiners are known to disagree when marking. We reportthe full battery of agreement statistics recommended in the psychometric and educational measurementliterature — Pearson correlation, mean absolute error (MAE), root-mean-square error(RMSE), exact and tolerance-band agreement, unweighted and weighted Cohen’s kappa, theintraclass correlation coefficient ICC(2,1), and Bland–Altman limits of agreement — and situatethe results against published estimates of human examiner-to-examiner reliability. GradeDrive’sagreement with AQA (quadratic-weighted κ = 0.885, ICC = 0.887, exact agreement = 77.5%)sits within the range reported for examiner-to-examiner agreement on comparable short-answeritem types. All 71 items, not only the disputed ones, were independently re-marked by a sixteacherpanel, which provides two further findings. First, on the 55 items where GradeDriveand AQA originally agreed, the panel’s marks matched the AQA/GradeDrive mark exactly on50 of 55 items (90.9%), with inter-teacher variation an order of magnitude smaller than on thedisputed items (mean SD 0.05 vs. 0.54 marks) — a validity check that supports both the paneland the “edge case” framing of the disputed items, even though a handful of the nominallynon-contentious items did show some scatter among teachers. Second, on the 16 items whereGradeDrive and AQA disagreed, GradeDrive’s marks were, on average, closer to the independentteacher consensus than AQA’s were (MAEGradeDrive = 0.69 vs. MAEAQA = 0.90 marks;GradeDrive closer on 9 of 16 items, AQA on 5, tied on 2), though a paired test on this smallsubsample does not reach conventional statistical significance (p = 0.39).