Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability

Abstract

Patients increasingly use large language models (LLMs) to seek medical information, but their performance in answering diabetic retinopathy (DR) questions remains insufficiently characterized. To evaluate the safety, accuracy, patient-centered communication, information quality, transparency, and readability of diabetic retinopathy information generated by widely accessible large language models. Fifty standardized DR patient questions were submitted to ChatGPT, Gemini, DeepSeek, Doubao, Copilot, and Perplexity. Responses were independently evaluated by five raters using guideline-based reference answer elements. Rater-based outcomes included Safety, Accuracy, Empathy, DISCERN, EQIP, JAMA benchmark criteria, and Global Quality Score (GQS). Readability was assessed using the Automated Readability Index, Coleman–Liau Index, Flesch–Kincaid Grade Level, Gunning Fog Index, SMOG, and Flesch Reading Ease Score. Inter-rater agreement and between-model differences were analyzed using appropriate agreement statistics and paired nonparametric tests. Inter-rater agreement was moderate for Safety (Fleiss’ κ = 0.581) and good for non-binary outcomes, with ICC (2,1) values ranging from 0.806 to 0.870. Most responses were classified as safe, with safe response rates ranging from 86.0 to 94.0%, and no significant between-model difference was observed. Accuracy scores were generally high across models, although between-model differences were statistically significant with a small effect size. Empathy varied more noticeably, with Copilot scoring highest and Doubao and Perplexity scoring lower. Perplexity achieved the highest DISCERN and JAMA scores, while Copilot performed best on EQIP. GQS differences were small. Readability differed significantly across models, with Gemini generally producing the most readable responses; however, all models exceeded the recommended sixth-grade reading level, and all Flesch Reading Ease Scores were below the recommended threshold. No single LLM performed best across all domains. Although the evaluated models generally provided safe and accurate responses to DR patient questions, limitations remained in safety escalation, empathy, transparency, and readability. LLM-generated DR information may support general patient education but should not replace professional ophthalmic consultation.