A cross-sectional evaluation of large language model chatbot interfaces for patient-facing herpes zoster information: safety, information quality, and readability
Sep 2026· Frontiers in Public Health· Vol 14· 0 citations· 61 references
Medicine
TL;DR
In this standardised benchmark, LLM chatbot interfaces provided generally favourable accuracy and information-quality scores but showed clinically relevant safety limitations, sparse source attribution and disclosure, and readability challenges.
Abstract
Background Large language model (LLM) chatbots are increasingly used to obtain health information. However, fluent and clinically plausible responses may still contain safety-relevant omissions, inadequate source attribution and disclosure, or difficult-to-read text. Objective To evaluate the safety, accuracy, empathy, information quality, response-level source attribution and disclosure, overall quality, and readability of five LLM chatbot interfaces answering lay-oriented questions about herpes zoster. Methods This cross-sectional evaluation used 46 standardised English-language questions across seven herpes zoster domains. Each question was submitted once to ChatGPT-5.5 Instant, DeepSeek-V4-Pro, Doubao-Seed-2.0-Pro, Gemini 3.5 Flash, and Qwen3.7-Plus under consumer-access conditions on June 2–3, 2026. Five dermatologists independently assessed safety, accuracy, empathy, DISCERN, Ensuring Quality Information for Patients (EQIP), Journal of the American Medical Association (JAMA) benchmark criteria, and the Global Quality Score (GQS). Six readability indices were calculated. Paired comparisons and inter-rater agreement were evaluated using prespecified statistical methods. Results The five interfaces generated 230 complete responses. Under the study conditions, 66 responses (28.7%) were classified as potentially unsafe using the prespecified ≥3/5-rater majority threshold, with no significant difference between interfaces (Cochran’s Q = 1.661, df = 4, p = 0.798). Sensitivity analyses using alternative thresholds showed consistent results. Accuracy and empathy differed across interfaces (both p < 0.001; Kendall’s W = 0.127 and 0.211, respectively), although effect sizes were small. Information-quality and overall-quality measures also showed outcome-specific differences. All six readability indices differed across interfaces (all p < 0.001); DeepSeek-V4-Pro generally produced text estimated to be easier to read, whereas Gemini 3.5 Flash produced text estimated to be more difficult to read. Safety agreement was substantial (Fleiss’ κ = 0.682), and ICC (2,1) values for other manually rated outcomes ranged from 0.832 to 0.889. Conclusion In this standardised benchmark, LLM chatbot interfaces provided generally favourable accuracy and information-quality scores but showed clinically relevant safety limitations, sparse source attribution and disclosure, and readability challenges. No interface consistently performed best across all outcomes. These findings represent single first responses obtained under the tested conditions and do not establish response stability across repeated queries. LLM-generated herpes zoster information should therefore be interpreted cautiously and should not replace individualised professional assessment or professionally reviewed patient information.
Across a limited set of standardized English-language prompts, the five chatbots often generated coherent HFMD information, but occasional safety-relevant inaccuracies and readability barriers remained, suggesting these systems may support general information seeking but should not replace individualized professional a...
Zhao-Le Gong, Yan Na, Yi Guo et al.· Frontiers in Public Health· 0 citations
Public-facing chatbots may support general patient education but should not replace individualized clinician-led prognostic communication as public-facing chatbots differed substantially in safety, reliability, communication quality, and readability.
Yan-Ru Jiang, Qian-Yun Wang, Liang Zheng et al.· Frontiers in Public Health· 0 citations
To compare the safety, medical accuracy, empathy, empathy, information reliability, educational quality, transparency, global quality, and readability of five large language model-driven chatbots when answering questions about pediatric vitamin D deficiency, a single-run evaluation is conducted.
Ping Shi, Tian Zhou, Qiao Nie et al.· Frontiers in Pediatrics· 0 citations
This exploratory study characterizes a recorded sample of specific chatbot responses, not stable or repeatable performance characteristics, and establishes a stable ranking of underlying model capability, clinical effectiveness, or suitability for unsupervised care.
Ling Miao, Li-Na Gu, Zhao-Le Gong et al.· Frontiers in Reproductive He...· 0 citations
Recurrent aphthous stomatitis (RAS), commonly known as oral ulcers, is a frequent and painful oral mucosal condition that often prompts people to seek consumer health information before professional consultation. Although LLM-generated health advice has been studied in several clinical fields, multidimensional evid...
Yu-Long Wang, Jing-Jing Li, Zhao-Le Gong et al.· Frontiers in Oral Health· 0 citations
Highlights What are the main findings? Instrument-based information-quality scores differed among the tested chatbot configurations, but these differences do not establish greater factual accuracy, guideline concordance, or clinical superiority; the full-set DISCERN comparison was limited because most questions were no...
Jian-Yu Chen, Zai-Han Zhao, Shang-Yu Han et al.· Healthcare· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Enforcing physics-based constraints prevents AI models from generating unreliable predictions for complex nonlinear systems in weather, climate, and turbulent flows.
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.