Erectile Dysfunction Management in the Age of Artificial Intelligence: Does ChatGPT-5 Provide Reliable and Accurate Evidence-Based Answers?
Abstract
Purpose
Erectile dysfunction (ED) is a highly prevalent but often under-reported and under-diagnosed condition. Many men are reluctant and embarrassed to seek proper medical care and invariably turn to online resources, including large language models such as ChatGPT for answers. We aimed to analyse ChatGPT responses to common ED questions.
Materials And Methods
Ten common ED questions were obtained using Google Trends and published guidelines. Questions were posed to ChatGPT-5. Responses were independently appraised by two authors using the Quality Evaluation Scoring Tool (QUEST). Two independent men's health experts then rated these answers for accuracy, completeness and actionability (0 to 9 scale). The Flesch-Kincaid system was used to assess readability. Inter-rater reliability was calculated using the intraclass correlation coefficient (ICC).
Results
Median QUEST score was 12/28 for both reviewers with low authorship and attribution scores. The highest scores (23 to 26) were for comorbidity and diagnostic testing questions. Inter-rater reliability for QUEST was excellent (ICC=0.92, 95% confidence interval 0.69 to 0.98). Expert ratings were high (median of 8/9 [range 5 to 9]) but demonstrated poor inter-rater reliability (ICC=-0.09), likely reflecting the restricted 0 to 9 scale and disproportionate effect of small differences in actionability rating. The correlation between mean QUEST and expert scores was weak (ρ=0.37, p=0.29), suggesting different dimensions of quality were captured. Readability was suboptimal with a mean Flesch-Kincaid of 47.3 (range 20.9 to 65.4) and a median grade level of 10 to 11.
Conclusions
ChatGPT can provide answers to common ED queries; however, it fell short of quality benchmarks. Difficult readability may be a barrier to some patients. While large language models have the potential to offer accessible patient information, further refinement and validation are required before such artificial intelligence can replace clinician counselling in the future.