“But it sounded confident”: the role of accuracy, tone, and disclaimers in users' medical decision-making
TL;DR
Accuracy emerged as the strongest predictor of user trust, although participants did not consistently identify inaccurate advice, and Conversational tone appears to have limited influence, while disclaimers may not consistently promote appropriate caution.
Abstract
Introduction Artificial intelligence (AI)-powered chatbots are increasingly used in healthcare for applications ranging from symptom triage to lifestyle guidance. Their effectiveness depends not only on their ability to provide reliable information but also on users engaging with their advice while remaining aware of potential inaccuracies. This study investigated how users perceive AI-generated medical advice, with a particular focus on the roles of accuracy, conversational tone, and disclaimers. Methods A survey was conducted with 115 participants who evaluated 15 chatbot scenarios representing five common health problems. Participants assessed chatbot responses varying in accuracy, tone, and the presence of disclaimers. Associations between these factors, user trust, willingness to seek second opinions, and participant characteristics (including age, health literacy, and prior chatbot use) were examined. Results Accuracy emerged as the strongest predictor of user trust, although participants did not consistently identify inaccurate advice. Individual characteristics, including age, health literacy, and prior chatbot experience, showed no significant associations with trust. Differences in chatbot tone had little influence on user perceptions, with chatbots generally being viewed as confident regardless of phrasing. The effects of disclaimers on willingness to seek second opinions were mixed and varied across scenarios, suggesting that disclaimers alone may not effectively prevent over-reliance on chatbot advice. Open-ended responses highlighted trust in healthcare professionals and the importance of clearly communicating AI limitations. Discussion This exploratory study, based on self-reported intentions in hypothetical scenarios, suggests that the accuracy of AI-generated medical advice is the primary determinant of user trust. Conversational tone appears to have limited influence, while disclaimers may not consistently promote appropriate caution. These findings indicate that healthcare chatbots should prioritize accuracy, clearly communicate their limitations, and encourage critical evaluation to achieve an appropriate balance between user trust and scepticism.