Evaluating the Accuracy, Empathy, and Readability of Generative AI Versus Registered Nurses in Discharge Planning: A Vignette‐Based Study

Abstract

ABSTRACT Aim To compare the multidimensional performance of discharge instructions generated by generative AI (GPT‐4) versus those created by clinical registered nurses across three dimensions—accuracy, empathy and readability—and to explore the impact of patient. Design A prospective, double‐blind, vignette‐based cross‐sectional study. Methods Five standardized multidisciplinary discharge scenarios were constructed. Discharge instructions were generated independently by five registered nurses and GPT‐4. Fifteen clinical experts conducted blinded assessments of accuracy, while 38 patients conducted blinded assessments of empathy and readability. Objective text features were extracted using natural language processing. Paired t‐tests or Wilcoxon signed‐rank tests were used to compare differences between groups, and a generalized linear mixed model was constructed to analyse factors influencing the acceptability of AI‐generated text. Results AI outperformed nurses in information comprehensiveness, but experts identified safety risks in AI‐generated texts, whereas no such issues were found in nurse‐produced texts. Nurses significantly outperformed AI in both empathy and readability, and objective NLP analysis confirmed that AI‐generated texts exhibited higher syntactic complexity and terminology density. The generalized linear mixed model indicated that advancing age and lower educational attainment were associated with reduced acceptance of AI‐generated texts. These findings derive from standardized vignettes under controlled experimental conditions and require further validation in real clinical settings. Conclusion Generative AI offers value as a drafting aid for ensuring information completeness in discharge instructions; however, its safety risks, empathy deficits, and linguistic complexity currently limit its standalone application. AI may be better positioned as an assistive tool for nurses rather than an independent communication tool, and its deployment should address potential digital divide issues among patient populations. Implications for the Profession and/or Patient Care These findings support positioning artificial intelligence as an information completeness tool requiring mandatory nurse review before patient delivery. Nurses remain essential for ensuring clinical safety, providing empathetic communication, and adapting language complexity to individual patient needs. Healthcare organizations should establish protocols requiring nurse verification of all artificial intelligence‐generated discharge content, with particular attention to medication dosages and contraindications. The identification of a digital divide necessitates that deployment strategies include health literacy assessment and tiered delivery approaches to prevent technological advances from exacerbating health communication inequalities among vulnerable populations. Impact What problem did the study address? ○ Generative artificial intelligence tools are increasingly proposed for clinical documentation, yet limited evidence exists comparing their discharge instruction quality against registered nurses—the professionals primarily responsible for discharge education—across multiple dimensions relevant to patient safety and comprehension. What were the main findings? ○ While artificial intelligence produced more comprehensive information, it generated clinically unsafe content and scored significantly lower than nurses in empathy and readability. Older and less‐educated patients showed reduced acceptance of artificial intelligence‐generated text, indicating a potential digital divide. Where and on whom will the research have an impact? ○ These findings inform nursing practice, healthcare informatics policy, and equitable care delivery globally, particularly regarding the safe integration of artificial intelligence tools into nurse‐led discharge education workflows for diverse patient populations. Reporting Method This study adhered to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines and incorporated principles from the Decision Support Systems Evaluation Guideline for Artificial Intelligence in Healthcare framework. No Patient or Public Contribution Patients and the public were not involved in the design, conduct, reporting or dissemination of this research. Trial and Protocol Registration This study is an observational cross‐sectional study and does not require clinical trial registration.