When AI Sounds More Helpful: Users’ Perceptions of AI-Generated and Physician-Provided Health Information
TL;DR
It is suggested that LLM-based chatbots may be perceived as useful supplemental information tools within future Internet health ecosystems, but their deployment should include safeguards that support transparency, verification, and appropriate reliance.
Abstract
AI-powered conversational agents are becoming part of the everyday Internet information ecosystem, reshaping how users seek, interpret, and act on health-related information outside clinical encounters. As large language model (LLM)-based chatbots are increasingly used as on-demand digital health information tools, understanding how users perceive their credibility, usefulness, and limitations is essential for the responsible design of future Internet-based health services. This mixed-method survey study examined how general adults evaluated healthcare-related question–answer pairs provided by physicians and generated by AI chatbots. A sample of U.S.-based adults recruited through Prolific (N = 62) rated each answer on clarity, usefulness, appropriateness of detail, trustworthiness, and perceived evidence, and provided open-ended explanations of their judgments. Primary mixed-effects analyses showed that both ChatGPT- and Claude-generated responses received higher overall participant ratings than physician-provided responses, although the estimated difference was substantially larger for Claude (ChatGPT–physician estimate = 0.250, 95% CI [0.135, 0.364]; Claude–physician estimate = 0.825, 95% CI [0.710, 0.939]). ChatGPT received higher ratings on four of the five dimensions but not on clarity, whereas Claude received higher ratings across all five dimensions. However, physician, ChatGPT, and Claude responses were always presented first, second, and third, respectively. Response source was therefore confounded with presentation position, and the observed differences cannot be attributed exclusively to source. The responses were also not matched for length or format. Qualitative findings showed that participants valued detailed, specific, and evidence-like explanations. Participants also expressed concerns about hallucination, privacy, over-reliance, and the need for clinician verification. These findings suggest that LLM-based chatbots may be perceived as useful supplemental information tools within future Internet health ecosystems, but their deployment should include safeguards that support transparency, verification, and appropriate reliance.