Evaluating the accuracy, reliability, and readability of AI chatbots in delivering postpartum depression information
TL;DR
AI chatbots, particularly ChatGPT-5, showed potential as supplementary tools for providing postpartum depression related information, especially in standardized MCQ based assessment, but inadequate transparency, limited source attribution, and suboptimal readability indicate that they should not be used as autonomous sources of postpartum mental health guidance and should not replace professional assessment or care.
Abstract
Background Postpartum depression (PPD) is a common perinatal psychiatric disorder with significant implications for maternal and infant health. Artificial intelligence (AI) chatbots have emerged as widely accessible tools for health information, but their performance in providing accurate, reliable, and readable PPD related information remains underexplored. Methods We evaluated six AI chatbots, ChatGPT-5, ChatGPT-4o, Claude Sonnet 4.5, DeepSeek-V3.2, DeepSeek-R1, and Gemini 2.5 Pro, using 200 standardized multiple choice questions (MCQs) on PPD to assess validity. ChatGPT-4o and DeepSeek-R1 were included only in the MCQ based validity analysis as earlier version comparators. Reliability and readability were further assessed using 20 core public education questions in the four latest models: ChatGPT-5, Claude Sonnet 4.5, DeepSeek-V3.2, and Gemini 2.5 Pro. Chatbot performance was assessed across three dimensions: validity, reliability, and readability. Each MCQ was presented three times independently, and each core public education question was assessed once per model. Results In the six model MCQ based validity analysis, ChatGPT-5 achieved the highest overall accuracy on MCQs (97.50% ± 0.50%). In the four models reliability and readability analyses, ChatGPT-5 obtained the highest DISCERN, EQIP, and GQS scores, suggesting relatively better content quality and user oriented usefulness. However, JAMA benchmark scores were low across all models, including ChatGPT-5, indicating limited transparency, source attribution, currency, and disclosure. All models produced outputs exceeding the recommended sixth grade reading level, although ChatGPT-5 and Gemini 2.5 Pro were relatively more accessible. Conclusion AI chatbots, particularly ChatGPT-5, showed potential as supplementary tools for providing postpartum depression related information, especially in standardized MCQ based assessment. However, this study did not evaluate clinical safety, patient comprehension, user behavior, or real world effectiveness, and suboptimal readability may limit accessibility for users with lower health or digital literacy. Inadequate transparency, limited source attribution, and suboptimal readability indicate that AI chatbots should not be used as autonomous sources of postpartum mental health guidance and should not replace professional assessment or care.