Integrating Generative AI Into Clinical Reasoning Education for Medical Students: Mixed Methods Study
TL;DR
A brief, theory-informed educational intervention integrating GenAI with clinical reasoning was associated with medium-to-large improvements in self-perceived AI literacy and collaborative attitudes, and this finding should be interpreted as inconclusive, given measurement and power constraints.
Abstract
Abstract Background Generative AI (GenAI) is increasingly integrated into clinical learning and practice. However, medical students often lack the competencies required for safe and critical use, including prompt design, output verification, and recognition of limitations. Educational interventions that integrate GenAI with clinical reasoning frameworks remain limited. Objective This study evaluated a structured, theory-informed workshop integrating GenAI, prompt engineering, and clinical reasoning education to enhance medical students’ self-perceived AI literacy and collaborative learning attitudes, and to assess whether patient-centered orientation changed following intensive AI exposure. Methods We conducted a single-group, pre-post, explanatory sequential mixed methods study with fifth-year medical students enrolled at an academic medical center between April 2024 and May 2025. The 3-hour workshop comprised 6 modules integrating clinical reasoning, cognitive-bias awareness, verification-oriented GenAI use, and hands-on prompt engineering and centered on ChatGPT (OpenAI). Quantitative outcomes were self-reported measures from a 20-item self-report AI-literacy questionnaire adapted from the Meta AI Literacy Scale and were examined with exploratory and confirmatory factor analysis, the 6-item Patient-Practitioner Orientation Scale-Short, and a modified Collaborative Learning Attitude Scale (CLAS). Pre-post change was assessed using 2-tailed paired t tests with Benjamini-Hochberg correction and Cohen d; the Patient-Practitioner Orientation Scale-Short and CLAS were available for a subsample (n=46). Qualitative data from 6 interviews and 17 reflective narratives were analyzed using reflexive thematic analysis and integrated with the quantitative findings. Results Among 150 eligible students, 139 (92.7%) completed paired AI-literacy assessments. Self-perceived AI literacy improved across all domains (Cohen d=0.69‐0.93, all P<.001; all remaining significant after false discovery rate correction), and collaborative learning attitudes increased substantially (d=0.94, P<.001). Patient-centered orientation showed no significant change (d=0.03); however, baseline scores were concentrated at the favorable end of the scale (a floor/restricted-range effect), and the subsample analysis was underpowered (minimum detectable dz=0.42), so this null result is inconclusive rather than evidence of unchanged orientation. Gains did not differ by sex or across the sequential cohorts. Reflexive thematic analysis (interviews: n=6; reflections: n=17) identified five themes and one emergent theme describing a shift toward verification-oriented GenAI use: (1) understanding GenAI capabilities and limitations, (2) prompt-engineering skill development, (3) calibrated trust through verification, (4) GenAI-supported communication and collaboration, and (5) ethical considerations, with emerging reconceptualization of professional identity. Conclusions A brief, theory-informed educational intervention integrating GenAI with clinical reasoning was associated with medium-to-large improvements in self-perceived AI literacy and collaborative attitudes. No detectable change in patient-centered orientation was observed; however, this finding should be interpreted as inconclusive, given measurement and power constraints. Embedding verification practices within clinical reasoning frameworks may offer a scalable approach for preparing physicians for responsible human-AI collaboration. Future studies should incorporate comparative designs, performance-based assessments, and longitudinal follow-up.