Quality and Readability of Four AI Chatbots Answering Search-Derived Patient Questions About Primary Aldosteronism: A Cross-Sectional Comparative Study

Abstract

Highlights What are the main findings? Instrument-based information-quality scores differed among the tested chatbot configurations, but these differences do not establish greater factual accuracy, guideline concordance, or clinical superiority; the full-set DISCERN comparison was limited because most questions were not treatment-focused. Responses from all four chatbots exceeded the prespecified sixth-grade readability threshold. What are the implications of the main findings? Chatbot responses should supplement, rather than replace, verified guidelines and individualized explanations from health professionals. Healthcare organizations should limit patient-facing chatbots to defined, low-risk uses. Organizations should also require source verification, plain-language output, human oversight, escalation advice, and monitoring after model updates. Abstract Background/Objectives: Patients use artificial intelligence (AI) chatbots for medical information, but response quality, source transparency, and readability for primary aldosteronism (PA) remain uncertain. This study compared four chatbot configurations and assessed whether their responses met patient-education reading levels. Methods: During a single-time-point snapshot on 27 July 2026, 17 questions derived from Medical Subject Headings and five-year worldwide Google Trends data were submitted once to GPT-5.5 through ChatGPT Plus, Copilot in Smart mode, Gemini 3.5 Flash, and the default Perplexity model, yielding 68 responses. DISCERN, the Ensuring Quality Information for Patients (EQIP) instrument, the Journal of the American Medical Association (JAMA) benchmarks, and the Global Quality Score (GQS) assessed structural information quality; six formulas assessed readability. These instruments did not assess statement-level factual accuracy. Differences were examined using Friedman tests and paired Wilcoxon signed-rank tests with Holm adjustment. Results: Scores differed for DISCERN (χ2 = 16.717, p < 0.001), EQIP (χ2 = 31.125, p < 0.001), JAMA (χ2 = 48.851, p < 0.001), and GQS (χ2 = 9.874, p = 0.020). Copilot had the highest median DISCERN and EQIP scores (43.00 and 75.00), followed by Perplexity (42.00 and 70.00). Median JAMA scores were 0 for ChatGPT and Gemini and 1 for Copilot and Perplexity. These instrument-based differences do not establish greater factual accuracy, guideline concordance, or clinical superiority; moreover, the full-set DISCERN comparison has limited interpretability because only two questions were explicitly treatment-related. Item-level kappa values ranged from 0.844 to 0.924, and total-score ICC(2,1) values ranged from 0.846 to 0.883. No readability measure met the sixth-grade benchmark. Conclusions: In these configurations, chatbots produced PA information. However, source transparency remained low, and language was overly complex. Relative score differences should not be interpreted as factual superiority. Patient-facing use requires verified sources, scope limits, human oversight, and routes to care.