Measuring Adolescents' Critical AI Literacy Levels : Based on Measurement Development Using Item Response Theory

Abstract

This study developed and validated a diagnostic tool to assess adolescents' critical AI literacy and analyzed its current status. Drawing on the framework of Veldhuis et al. (2025), the researchers constructed items across four dimensions and finalized them through expert review and validation. Data from 1,010 middle and high school students were analyzed using Item Response Theory (IRT), revealing that the four-factor, two-parameter logistic (2PL) model provided the best fit. The tool demonstrated particular effectiveness in discriminating adolescents with average to below-average critical AI literacy levels. By domain, adolescents achieved high correct response rates with minimal variance in “Taking Action,” while they were most challenged by “Interrogating Multiple Viewpoints,” which requires the comparison and evaluation of diverse information sources and perspectives. Furthermore, substantial individual variance was observed in the “Focusing on Sociopolitical Issues” competency. These findings inform recommendations for critical AI literacy education that prioritize the analysis and practice of sociopolitical contexts within AI mechanisms and outputs.