Skip to content

Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

Aug 2026 · arXiv.org · Vol abs/2608.15630 · 1 citation · 56 references
Computer Science

TL;DR

Analytical approach combines exploratory factor analysis, factor congruence, and resampling to assess latent structure similarity across human learners and LLMs, finding systematic differences between human and LLM factor structures.

Abstract

The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities. A common approach is to evaluate LLMs using assessment instruments originally designed to measure skills and competencies in humans, such as standardized exams, and to use performance on these instruments as evidence for generalizable claims about LLMs'underlying abilities on the same skills the assessments are intended to measure in humans. However, from a validity perspective, such inferences require that the relationship between observed performance and underlying constructs established for humans also holds for LLMs. In particular, a necessary condition for transferring score interpretations is similarity in the latent structure of responses to the assessment. In this study, we examine whether this condition holds in two educational contexts: high-school chemistry and a quantitative reasoning section of a university entrance exam. Using a case study design, we compare human response data with responses generated by six multimodal LLMs. Our analytical approach combines exploratory factor analysis, factor congruence, and resampling to assess latent structure similarity across human learners and LLMs. Across both instruments, we find systematic differences between human and LLM factor structures, showing evidence that the analyzed assessments may not measure the same constructs for humans and LLMs. These findings call into question the validity of evaluation practices that use educational assessments to make claims about AI capabilities.

View source

Similar papers

Open access Oct 2026

Evaluating Large Language Model–Derived Outputs as Psychological Measures: A Five-Domain Psychometric Framework With Application to Daily Stress

Abstract Background Large language models (LLMs) are increasingly used to quantify psychological constructs from natural language, yet their outputs are typically evaluated only as predictions of an external label. When that label is itself imperfect, optimizing prediction conflates label agreement with construct measu...

Inyong Jeong, Soohyun Park, M. Jhon et al. · 0 citations
Preprint Oct 2026

Diagnosing the Explicit-to-Applied Theory of Mind Gap in LLMs

As large language models are deployed in high-stakes roles such as therapy, education, and personal assistance, their social cognition must be assessed reliably. Current Theory of Mind (ToM) benchmarks report aggregate scores that conflate two distinct competencies: explicit ToM (recognizing what someone believes) and...

Prajvi Saxena, Sabine Janzen, W. Maass · 0 citations
Sep 2026

Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence

This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and...

Satoshi Takahashi, A. Yoshikawa, Megumi Kose et al. · 0 citations
Review Sep 2026

A Benchmark for LLM's Understanding of Middle School and High School Science Topics

Large language models (LLMs) are increasingly integrated into educational settings, yet educators lack robust, standards-aligned tools to evaluate their effectiveness in K-12 science contexts. Existing benchmarks predominantly assess general language or advanced scientific reasoning, leaving a critical gap in understan...

Noah L. Schroeder, Yessy Ambarwati, Yu-Ji Zhang et al. · 0 citations
Conference Open access 2026

Automated Assessment of Children’s Writing Competence Using Large Language Models

: This study examines the validity of DeepSeek-R1 for the automated assessment of children's writing competence in Spanish. We analyzed compare-contrast texts produced by 126 4th-grade students from three primary schools in Spain across three analytic dimensions: Structure, Coherence, and Quality. The evaluation spanne...

Alejandro Alvarez-Varela, Haithem Afli, Marisol Cueli et al. · 0 citations
Review Open access Nov 2026

Item response theory in educational performance assessment: a review of models, challenges, and emerging directions in education

Performance-based assessment has become central to modern education, particularly in domains that emphasize practical skills, creativity, and problem-solving. Unlike traditional tests, these assessments rely on authentic tasks, complex rubrics, and human judgment, raising concerns about fairness, reliability, and valid...

T. Balla, S. Király, Á. Németh · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.