Artificial intelligence versus physician decision-making in acute heart failure: a comparative evaluation of four contemporary AI systems
Abstract
Abstract Aims Large language models (LLMs) are increasingly used by the public and healthcare professionals for clinical reasoning. However, they were developed as general-purpose language models rather than dedicated clinical decision-support tools, and their performance in real-world acute heart failure (AHF) management remains insufficiently evaluated. We compared four contemporary AI models with physician decision-making. Methods and results This single-centre retrospective observational study incorporated a cross-sectional AI evaluation of consecutive patients treated at Sotiria General Hospital between July and November 2025, excluding acute coronary syndrome. ChatGPT, ChatGPT Plus, DeepSeek, and an ESC guideline-based retrieval-augmented generation (RAG) model independently assessed identical standardized cases. Recommendations were compared with physician decisions on diagnosis, treatment, guideline-directed medical therapy, timing, and prognosis using Cohen’s kappa, McNemar’s test, Cochran’s Q, Wilcoxon signed-rank testing, and multivariable logistic regression. Diagnostic performance differed significantly across systems (Cochran’s Q = 8.477, P = 0.037). In the balanced diagnostic cohort, ChatGPT Plus achieved the highest heart failure (HF) accuracy (accuracy 86.5%, sensitivity 87.0%, and specificity 86.0%), followed by RAG (accuracy 77.5%), ChatGPT (accuracy 77.0%), and DeepSeek (accuracy 69.0%). For HF within the top three differential diagnoses, sensitivity was 90.0–97.0%; specificity was 70.0% for ChatGPT Plus, 68.0% for RAG, 19.0% for DeepSeek, and 15.0% for ChatGPT. Physician agreement was highest for guideline-directed therapies, with greater variability in prognosis and escalation of care. RAG more closely matched physician therapeutic timing but did not consistently outperform general-purpose models. Conclusion Contemporary AI models show promise in AHF decision support but substantial task-specific variability, supporting use as complements rather than replacements for physician judgement.