Evaluating the constructiveness and professional tone of AI-generated corrective feedback for medical learners

Abstract

Faculty feedback is vital for the professional growth of learners. Barriers such as faculty’s confidence in their feedback skills and time limitations may inhibit the provision of quality feedback. Generative large language models (LLMs), such as ChatGPT, have the potential to support faculty in providing high-quality corrective feedback by producing effective narrative feedback scripts for faculty to reference. This study evaluates the constructiveness and professional tone of Artificial Intelligence (AI)-generated corrective feedback scripts, compared with educator-authored scripts, across standardized learner deficiency scenarios. Six pediatric medical education leaders blindly rated two narrative feedback scripts for eighteen learner deficiency scenarios; one script for each scenario was authored by educators, the other script was generated by ChatGPT without editing. The raters scored each script in four domains (behavior-centric, balanced between positive and negative, non-judgmental, and focused) using a 3-point scale (0–2), where 0 = No, 1 = Somewhat, 2 = Yes. The median scores of the educator authored scripts and the ChatGPT-generated scripts were compared by domain. An overall median score was also generated for the educator authored scripts and the ChatGPT-generated. Combining all four measured domains, ChatGPT-generated scripts received higher combined median scores, had increased odds of receiving higher domain scores than educator authored scripts, and were preferred by raters for 78% of the scenarios. AI-generated feedback scripts were rated more favorably for constructiveness and professional tone across four assessed domains suggesting that they may have potential as an adjunct to faculty feedback, however, these findings are limited to communication characteristics and do not address accuracy, educational impact, or clinical applicability. AI may serve as a supportive drafting tool, but further study is needed before conclusions about real-world utility can be made.