2026· International Conference on Language Resources and Evaluation· pp. 281-298· 2 citations· 45 references
Computer Science
TL;DR
This work investigates the performance of various LLM-as-a-judge in a low-resource scenario, namely Basque, and evaluates its correlation with human judgements, and shows that state of the art models exhibit fairly poor correlations with humans and amongst themselves.
The results show that high self-consistency does not necessarily indicate high agreement with human judgments when using local LLMs as automatic judges, and highlight the need to evaluate both consistency and human alignment when using local LLMs as automatic judges.
EvalAI grades theory answers on meaning rather than wording, using Sentence-BERT to measure how semantically similar a student's response is to a lecturer-provided reference answer, showing that semantic grading of this quality can be delivered within a deployable web application without specialised machine learning in...
E. Mgbeahuruike, Chris-Esezobor Ejodamen, Nelson-Nwanoneze Samuel et al.· British journal of computer,...· 0 citations
This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and...
Satoshi Takahashi, A. Yoshikawa, Megumi Kose et al.· arXiv.org· 0 citations
Although LLM-based benchmark ratings can approximate expert judgments and reduce the need for labor-intensive human triple coding, limitations remain regarding cost, genre- and task specificity, and sensitivity to text presentation and student grade level—factors that constrain immediate classroom use, particularly for...
LLJ Cards is introduced, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations and provides a structured framework for applying these principles in the design and reporting of automated evalua...