Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful reading of accompanying dataset documentation. We present the first benchmark enabling end-to-end evaluation of metadata extraction aligned with a community-standard schema....
Berke Arda, Ahmetcan Yavuz, Paul Gerry et al.· 0 citations
Large language models (LLMs) trained to answer questions are natively poor at teaching. Reinforcement Learning (RL) against a simulated student is a promising approach to improve their pedagogy, but existing RL-trained tutors reward the student's success on the tutored problem with the tutor's words still in context. T...
Jakub Macina, Manu Kapur, Mrinmaya Sachan· 0 citations
Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of th...
Peng Cui, Qiao-Yuan Zheng, R. Debelak et al.· 0 citations
This work examines how large language models reason about misconceptions when generating distractor answers for multiple-choice questions (MCQs) and introduces a taxonomy over reasoning strategies for distractor generation that is grounded in learning-science literature and empirical observation.
Yanick Zengaffinen, Andreas Opedal, Donya Rooein et al.· arXiv.org· 2 citations
Human knowledge is inherently structured and interdependent: mastery of a concept requires prior mastery of its prerequisites, a principle formalized by Knowledge Space Theory (KST). While LLMs achieve strong performance on complex reasoning tasks, it remains unclear whether they exhibit coherent, human-like knowledge...
The Last Translation Benchmark is introduced, a collection of human-authored and peer-reviewed examples that break leading machine translation models and a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and a...
Vilém Zouhar, Niyati Bafna, Mukund Choudhary et al.· 0 citations
This work evaluates prompting, supervised fine-tuning (SFT), and reinforcement learning (RL) methods for writing simulation using automated essay scoring as a measure of profile alignment and suggests that explicit supervision enables substantially stronger profile alignment than prompting alone.
Heejin Do, Jakub Kontak, Mrinmaya Sachan· 1 citation
This work formalizes multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model, and proves the optimality of the proposed algorithms and shows that it improves discrimination between top-performing m...
Vilém Zouhar, Julia Kreutzer, A. Lavie et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.