Quality and readability of large language model–generated responses on type 2 diabetes remission: a cross-sectional balanced factorial study

Abstract

Abstract Large language models (LLMs) are increasingly being explored for patient education and medical information delivery. However, their outputs may involve trade-offs among medical accuracy, comprehensiveness, empathy, and readability. Type 2 diabetes remission is a clinically nuanced concept that is readily misunderstood by patients, making it a suitable context in which to evaluate both the medical quality and patient accessibility of artificial intelligence–generated health information. To systematically compare the medical quality and readability of responses concerning type 2 diabetes remission generated by different LLMs and prompting strategies, and to characterize their joint quality–readability profiles. We conducted a cross-sectional study using a balanced factorial design. Responses were generated for 16 core questions concerning type 2 diabetes remission using 4 LLMs—DeepSeek-V3.2,GPT-5.5,Gemini 3.1 Pro and Claude Sonnet 4.6—and 6 prompting strategies: Keyword, Question, 6th-grade, Guideline, Patient, and Doctor. This yielded 384 AI-generated responses. Two endocrinologists, each with more than 5 years of clinical experience, independently evaluated medical quality using a prespecified rubric. Five domains—Accuracy, Relevance, Comprehensiveness, Empathy, and Falsehood avoidance—were rated on 5-point ordinal scales. A descriptive Composite quality score was also calculated. Readability was assessed using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFOG), and Simple Measure of Gobbledygook (SMOG). Quality ratings were analyzed using cumulative link mixed models, whereas readability outcomes were analyzed using linear mixed-effects models; Question ID was included as a random intercept in all models. Exploratory analyses examined Spearman correlations between readability and quality and identified Pareto-efficient model–prompt combinations. All 384 responses were included in the analysis, with no missing data. The descriptive Composite quality score ranged from 4.175 for Gemini 3.1 Pro to 4.285 for DeepSeek-V3.2 across models and from 4.066 for the 6th-grade strategy to 4.350 for the Doctor strategy across prompting strategies. Cumulative link mixed models showed significant main effects of model across all 5 quality domains: Accuracy (LR χ² = 10.964, LR-based pseudo-R² = 0.0281, 95% CI 0.0015–0.0636, P = 0.012), Relevance (LR χ² = 54.042, LR-based pseudo-R² = 0.1313, 95% CI 0.0688–0.1974, P < 0.001), Comprehensiveness (LR χ² = 12.018, LR-based pseudo-R² = 0.0308, 95% CI 0.0025–0.0677, P = 0.007), Empathy (LR χ² = 50.905, LR-based pseudo-R² = 0.1242, 95% CI 0.0632–0.1890, P < 0.001), and Falsehood avoidance (LR χ² = 18.752, LR-based pseudo-R² = 0.0477, 95% CI 0.0107–0.0922, P < 0.001). Prompting strategy also showed significant main effects across the quality domains, with the largest effect observed for Empathy (LR χ² = 243.836, LR-based pseudo-R² = 0.4701, 95% CI 0.3786–0.5477, P < 0.001). Descriptively, GPT-5.5 had the numerically highest mean scores for Accuracy (4.448) and Falsehood avoidance (4.469), whereas DeepSeek-V3.2 had the numerically highest mean scores for Relevance (4.948), Comprehensiveness (4.427), and Empathy (3.615). Among prompting strategies, Patient had the numerically highest mean Empathy score (4.359), whereas Doctor had the numerically highest mean Comprehensiveness score (4.609). For readability, both model and prompting strategy showed significant main effects on FRES, FKGL, GFOG, and SMOG. DeepSeek-V3.2 had the numerically highest mean FRES (51.28), whereas Claude Sonnet 4.6 had the lowest (32.98); among prompting strategies, 6th-grade had the numerically highest mean FRES (69.70), whereas Doctor had the lowest (27.50). In an exploratory response-level correlation analysis that did not account for clustering within Question ID and was intended for hypothesis generation only, FRES was positively correlated with Empathy (ρ = 0.53) and negatively correlated with Comprehensiveness, Falsehood avoidance, and the Composite quality score (ρ = −0.46, − 0.40, and − 0.26, respectively; FDR-adjusted q < 0.001), indicating directionally divergent association patterns between readability and different dimensions of medical quality. In patient-education responses related to type 2 diabetes remission, LLM output performance was influenced by both model choice and prompting strategy. No single model–prompt combination consistently outperformed the others across all evaluated outcomes. Within the T2DM remission question set examined in this study, response optimization required trade-offs among accuracy, comprehensiveness, readability, falsehood avoidance, and empathy.