The Scale for AI Literacy in Health Care Workers: Development and Validation
TL;DR
The SAIL-HCW provides initial evidence of validity and reliability for assessing AI literacy among health care workers and suggests that AI literacy may be represented as a general construct with additional domain-level components.
Abstract
Abstract Background AI is increasingly embedded in health care systems; yet, validated instruments for assessing AI literacy among health care workers remain limited. Existing measures are often designed for students or general populations and may not adequately reflect competencies required in health care practice. Objective This study aimed to develop and validate the Scale for AI Literacy in Health Care Workers (SAIL-HCW), a new instrument designed to assess AI literacy across domains relevant to health care practice. Methods A 3-phase instrument development study was conducted. In Phase 1, conceptual domains were identified through a literature review, and an initial item pool was generated. In Phase 2, content validity was assessed by 4 subject-matter experts, and face validity was evaluated with 26 health care workers. Feedback from both groups informed item refinement. In Phase 3, psychometric testing was conducted using survey data from health care workers in a single health care organization. A total of 425 participants completed the survey. The dataset was randomly split into 2 subsamples for exploratory factor analysis (n=212) and confirmatory factor analysis (n=213). Model fit was evaluated using unidimensional, correlated-factor, higher-order, and bifactor models. Reliability was assessed using Cronbach alpha and McDonald omega. Item performance was examined using corrected item-total correlations (CITC), item discrimination analysis, and inter-item correlations. Construct validity was assessed using prior AI training, frequency of AI use, and self-rated AI literacy. Results Phase 2 feedback from experts and health care workers supported the proposed domain structure and informed item refinement, including revision of wording and removal of redundant items. The final SAIL-HCW consists of 14 items across 7 domains, including AI concept, data fluency, AI evaluation, AI in practice, ethics and regulation, AI in system, and continuous learning. In Phase 3, the bifactor model showed the best fit compared with alternative models (comparative fit index and Tucker-Lewis index>0.93; root-mean-square error of approximation<0.06; standardized root-mean-square residual<0.05), indicating a general AI literacy factor alongside domain-specific factors. Internal consistency for the total scale was high (Cronbach α=0.937; ω=0.938). Domain-level reliability ranged from 0.635 to 0.797. All items significantly discriminated between high- and low-scoring groups (P<.001), with CITC values ranging from 0.570 to 0.785. Construct validity was supported, with higher SAIL-HCW scores observed among participants with prior AI training, higher frequency of AI use, and higher self-rated AI literacy (all P<.001). Conclusions The SAIL-HCW provides initial evidence of validity and reliability for assessing AI literacy among health care workers. Findings suggest that AI literacy may be represented as a general construct with additional domain-level components. The scale may be useful for research and educational evaluations, although further validation in other settings is required.