Automatic Evaluation of Handwritten Answer Scripts Using Deep Learning

Abstract

Manual evaluation of handwritten exams is time-consuming and inconsistent for educational institutions. Teachers spend considerable effort on grading, which affects the delayed result delivery and heterogeneity of evaluation criteria. Despite numerous attempts made previously to solve the problem using automated solutions, they frequently lacked satisfactory handwriting recognition accuracy, were unable to process multiple questions in one paper, or depended on expensive commercial software. Therefore, development of an automated system capable of properly evaluating handwritten papers becomes increasingly important for ensuring efficiency and fairness of educational testing. Against this background, the current study is aimed at creation of the system for automated analysis of answers and their categorization depending on their quality. For the purposes of this project, TrOCR model was used as a handwriting recognition system with an accuracy of 91.8% of recognized characters. Two types of parsing can be selected: either Llama3, based on large language models, and suitable for recognizing complex texts or regular expressions parser. The grading algorithm combines five different similarity measures and adjusts their coefficients according to the obtained score. As a result of experiments, the Pearson correlation between automatic and human expert grading of 0.912 and R-squared of 0.832 were achieved, with a mean absolute error equal to 6.18 points. Responses are processed within 8.5 seconds using GPU, which makes the developed tool 97% faster than manual grading and does not require any license fees.