Psychometric Validation of LLM-Generated Items: Equivalent Quality to Human-Authored Assessments via IRT and Learning Analytics

Abstract

Personalized learning environments rely on a large number of high-quality assessment items, yet manually authoring such items is costly. Large language models (LLMs) demonstrate strong generative capabilities; however, empirical evidence regarding whether LLM-generated items possess sufficient psychometric quality and effectively support learning remains limited. This study introduces an integrated evaluation framework connecting LLM-based item generation, IRT-based psychometric inspection, and analysis of learning behaviors and performance. The framework was validated in a university-level Computer Vision course using a GPT-4o system enhanced with Retrieval-Augmented Generation (RAG). Within this specific course context, the results show no significant differences in the difficulty, discrimination, and Test Information Function (TIF) of AI-generated items compared to human-authored items, demonstrating stable measurement quality. Furthermore, students’ correct rate in practice was significantly positively correlated with their academic achievement, indicating a beneficial learning impact. This work offers context-specific empirical evidence on the psychometric properties and learning effectiveness of LLM-generated items, laying a foundation for adaptive and personalized learning environments. However, further validation across different subject domains remains necessary.