An Exploratory Study on the Development of a Student AI Literacy Diagnostic Tool Based on the OECD AI Literacy Framework

Abstract

Although AI literacy is increasingly recognized as a fundamental competence for all learners, field-applicable diagnostic tools for systematically measuring K–12 students' AI literacy remain insufficiently developed. This foundational-stage study developed an AI literacy diagnostic tool grounded in the OECD AI Literacy Framework (2025) and explored its applicability through field implementation in Korean primary and secondary education settings. To develop the tool, an expert review was conducted. The tool was then administered to middle school students at digital-based education innovation schools in Region C, with online surveys conducted in August (pre-test) and November (post-test) 2025, from which internal consistency reliability, paired-sample t-tests, and effect size analyses were computed. As a result, a diagnostic tool comprising 25 items across five domains was developed, and the expert review confirmed preliminary content validity, with a mean item-level validity rating of 4.50 and mean ratings of 4.88 for content validity, explanatory power, and usefulness (out of 5). Cronbach's  computed from pre-test data (N=2,583) ranged from .867 to .918 across domains, with a total scale alpha of .969, indicating strong internal consistency. Pre- and post-test analysis of matched middle school samples (N=796) confirmed statistically significant improvements across all five AI literacy domains, with 21 of 25 items showing significant change. As a foundational-stage study preceding confirmatory factor analysis or structural equation modeling, subsequent rigorous psychometric validation and controlled effectiveness studies are warranted.