Skip to content
Review

Abstract 14187: Development and Pilot Evaluation of a Large Language Model-Powered Self-Management Chatbot Application for Heart Failure

Nov 2026 · Circulation · 0 citations

Abstract

Introduction: Heart failure (HF) is associated with high morbidity and healthcare utilization. Large language model (LLM)-enabled tools may provide continuous, scalable support for self-management of HF, but evidence on quality and safety is lacking. Objective: Evaluate an LLM-powered HF self-management chatbot application (app), CardioCoach AI, on quality, safety, and clinical reasoning in simulated patient interactions. Methods: CardioCoach AI contains a multimodal (voice and text), chat-based interface and a trained LLM back-end system which: (1) provides proactive daily symptom check-ins and medication reminders, (2) offers customized, responsive education, and (3) sends summary reports and automated alerts to care teams in response to high-risk symptoms. The app contains critical safety features, including a retrieval-augmented generation (RAG) architecture grounded in validated HF educational materials to minimize inaccurate or fabricated responses, and a rule-based triage logic to assess reported symptoms as stable, potentially urgent, or definitely urgent. To evaluate the app’s quality, safety, and clinical reasoning, three HF clinical experts independently evaluated 10 simulated patient interactions with the chatbot (3 triage, 7 education scenarios; see example interaction in Figure 1). The reviewers with cardiology expertise used a 15-item questionnaire, adapted from “Questionnaire for User Interaction Satisfaction and Evaluation of Technology” (QUEST), to score interactions across five domains (i.e., information quality, safety, clinical reasoning accuracy, communication persona, and trustworthiness). We calculated scores for each subdomain and summed them to create an overall quality score, reporting the median and interquartile range (IQR) of scores across the expert reviewers. Results: The QUEST-based evaluation scores are shown in Figure 2. The median (IQR) overall quality was 88% (80-91%). Information quality (92% [58-92%]) and safety (92% [83-100%]) scored highest. Clinical reasoning and communication persona both scored 83% (83-100%), and trustworthiness scored 78% (78-89%). All triage scenarios adhered to predefined logic rules, but scores for the interaction in the potentially urgent triage scenario were lowest (47% [40-70%]). Conclusions: CardioCoach AI demonstrated strong performance across quality and safety domains in simulated interactions, with opportunities for future system refinement around potentially urgent triage scenarios.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.