Development and Validation of an AI-Era Learning Competency Assessment for College Students

Abstract

This study aimed to validate the practical components of learning competency by reflecting changes in the university learning environment resulting from the recent expansion of digital environments and artificial intelligence (AI). To achieve this purpose, preliminary items consisting of three dimensions—learning cognition, learning skills, and learning attitudes—and nine subdomains, including items measuring AI-integrated learning performance, were developed and their content validity was confirmed through an expert Delphi survey. The preliminary version of the Learning Competency Diagnostic Test (H-LCT(II)) was administered to 258 undergraduate students at H University. The collected data was analyzed through exploratory factor analysis (EFA), confirmatory factor analysis (CFA), construct validity verification, and competitive model comparisons to establish the validity of the instrument from multiple perspectives. The results indicated that the nine subdomains formed a bifactor hierarchical structure consisting of a learning dimension and an AI-utilization dimension, and the validity of the structure was empirically supported at multiple levels. In addition, the reliability, convergent validity, and discriminant validity at the latent construct level all satisfied the recommended criteria. Consequently, a final measurement model consisting of 41 items across the learning and AI-utilization dimensions was confirmed. The Learning Competency Diagnostic Test (H-LCT(II)) developed in this study is expected to be useful for diagnosing university students’ AI-era learning competencies and for establishing educational interventions and policy support at the university level.