Evaluating Kidney Cyst Information from Chatbots: A Comparative Study

Abstract

Objective: Kidney cysts are the most common acquired lesions of the kidney. This study aims to assess the reliability and quality of responses provided by three different artificial intelligence chatbots to questions related to kidney cysts.Materials and Methods: Using the Semrush platform, we analyzed kidney cyst-related search trends over five years. We examined 25 questions, categorized into four subgroups: “general information”, “symptoms and diagnostic methods”, “treatment and follow-up procedures”, and “side effects and complications”. These questions were posed to three chatbots (ChatGPT-4, Gemini Pro, Llama 3.1 Large). Response readability was assessed using Flesch-Kincaid Grade Level (FKGL) test and Flesch Reading Ease Score (FRES) test. Reliability and quality were evaluated using the Ensuring Quality Information for Patients (EQIP) score and modified DISCERN score.Results: Llama had a significantly higher FKGL score than ChatGPT (p = 0.022), with no significant difference between Llama and Gemini (p = 0.500), or Gemini and ChatGPT (p = 0.525). On the FRES scale, ChatGPT scored higher than Gemini, and Gemini outperformed Llama (p = 0.006 and p = 0.012, respectively). The median modified DISCERN score was lowest for ChatGPT (2), compared to Gemini (3) and Llama (3) (p < 0.001). The mean EQIP score was 71.22 ± 3.91 for ChatGPT, 77.72 ± 1.16 for Gemini and 73.26 ± 2.70 for Llama (p < 0.001). ChatGPT had significantly lower quality scores than the other two chatbots.Conclusion: Although ChatGPT generated the most readable responses according to the FRES score, it demonstrated lower quality and reliability than Gemini and Llama. Overall, chatbot responses remained challenging in terms of readability and comprehension, highlighting the need for further improvements before they can be considered a reliable standalone source of patient education.