Skip to content
Conference Open access

Automated Assessment of Children’s Writing Competence Using Large Language Models

2026 · Proceedings of the 22nd International Conference on Web Information Systems and Technologies · 0 citations · 22 references

Abstract

: This study examines the validity of DeepSeek-R1 for the automated assessment of children's writing competence in Spanish. We analyzed compare-contrast texts produced by 126 4th-grade students from three primary schools in Spain across three analytic dimensions: Structure, Coherence, and Quality. The evaluation spanned five distinct measurement phases. Automated scores generated by the Large Language Model were compared against human expert consensus using the Intraclass Correlation Coefficient model (3,1), Spearman rho, Mean Absolute Error percentage, and Wilcoxon signed-rank tests with Holm-Bonferroni correction. Intraclass Correlation Coefficient values were uniformly poor across all 15 phase-by-dimension combinations (range: 0.128 – 0.443). This performance was driven by a systematic negative bias that emerged from the first test onwards and peaked at the second test, with an underscoring of -17 to -25 points per dimension. Conversely, Spearman rho values improved substantially over time, rising from 0.23 – 0.31 at Pretest to 0.45 – 0.63 at Posttest and Follow-Up. This dissociation between rank ordering and absolute calibration indicates that the current pipeline cannot follow students' score gains during instruction. Therefore, the system is unsuitable for longitudinal growth tracking without phase-specific recalibration. Baseline screening remains the most viable near-term application.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.