Skip to content

Judging Instruction Responses in a Low-Resource Language: A Case Study on Basque

2026 · International Conference on Language Resources and Evaluation · pp. 281-298 · 2 citations · 45 references
Computer Science

TL;DR

This work investigates the performance of various LLM-as-a-judge in a low-resource scenario, namely Basque, and evaluates its correlation with human judgements, and shows that state of the art models exhibit fairly poor correlations with humans and amongst themselves.

View source

Similar papers

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

The results show that high self-consistency does not necessarily indicate high agreement with human judgments when using local LLMs as automatic judges, and highlight the need to evaluate both consistency and human alignment when using local LLMs as automatic judges.

Aakash Kumar Tiwari · 2 citations
Open access Sep 2026

Design and Implementation of a Sentence-BERT-Driven Semantic Matching Model for Automated Evaluation of Theory Answers

EvalAI grades theory answers on meaning rather than wording, using Sentence-BERT to measure how semantically similar a student's response is to a lecturer-provided reference answer, showing that semantic grading of this quality can be delivered within a deployable web application without specialised machine learning in...

E. Mgbeahuruike, Chris-Esezobor Ejodamen, Nelson-Nwanoneze Samuel et al. · 0 citations
Sep 2026

Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence

This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and...

Satoshi Takahashi, A. Yoshikawa, Megumi Kose et al. · 0 citations
#small language model Open access Sep 2026

Human versus machine

Although LLM-based benchmark ratings can approximate expert judgments and reduce the need for labor-intensive human triple coding, limitations remain regarding cost, genre- and task specificity, and sensitivity to text presentation and student grade level—factors that constrain immediate classroom use, particularly for...

Afra Sturm, Valentin Unger, Fabian Grünig · 0 citations

LLJ Cards: Best practices for the Use of LLMs as Judges

LLJ Cards is introduced, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations and provides a structured framework for applying these principles in the design and reporting of automated evalua...

Khaoula Chehbouni, Melina Medjdoub, Florian Carichon et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.