It is concluded that stronger evaluation practices and closer collaboration with clinicians are needed to produce models that are clinically useful and safe to deploy.
Abstract
Artificial intelligence is rapidly transforming modern medicine, providing powerful tools to assist clinicians. However, the most accurate models are opaque, and their limited interpretability hinders clinical adoption and raises regulatory concerns. Explainable Artificial Intelligence (XAI) aims to make these models’ decisions comprehensible. Despite its rapid growth and the many reviews already published, aspects such as the rigor of explanation evaluation, clinician involvement, and the use of large language models to generate explanations appear to have received limited attention. This article is a structured, relevance-prioritized review of XAI applied in healthcare, organized around six research questions covering the XAI techniques used, the data and clinical domains they are paired with, the models they explain, how evaluation is performed, the identified limitations, and the use of large language models. Unlike prior reviews that each address only part of this scope, we bring these dimensions together and quantify how rigorously explanations are evaluated and how often the underlying models are externally validated. We conducted a structured search across four literature sources: IEEE Xplore, PubMed, Google Scholar, and PubMed Central (PMC). We adapted the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow by carrying a relevance-prioritized subset forward to full-text eligibility assessment, yielding 64 articles for synthesis. Within this corpus, our review revealed a predominantly image-based, post hoc XAI landscape dominated by a few established methods such as SHapley Additive exPlanations (SHAP), Gradient-weighted Class Activation Mapping (Grad-CAM), and Local Interpretable Model-agnostic Explanations (LIME), mostly paired with convolutional, tree-based, and hybrid models on classification and risk-prediction tasks. Clinicians evaluated the explanations in only a minority of the included studies, and relatively few models were externally validated, with no evidence of large language models in our corpus. We conclude that stronger evaluation practices and closer collaboration with clinicians are needed to produce models that are clinically useful and safe to deploy.
A narrative review of explainable artificial intelligence (XAI) in healthcare, specifically medical diagnosis, based on a targeted selection guided by a narrative review of recent literature, from 2019 to 2026, conducted on Scopus, Web of Science, and PubMed databases.
Guilherme Barbosa, Eduardo Carvalho, D. Oliveira et al.· Diagnostics· 1 citation
Large language models such as GPT-4, Med-PaLM 2, and BioMedLM are being increasingly applied to clinical decision support tasks including diagnostic reasoning, clinical documentation, medical question answering, and patient-facing health communication, with several studies reporting performance comparable to trained cl...
M. Oloko-Oba, Ebenezer Esenogho, Kehinde D. Aruleba et al.· Applied Informatics· 0 citations
Artificial intelligence (AI) systems are increasingly studied for tasks such as medical-image analysis, risk prediction, and clinical decision support. However, strong predictive performance does not by itself show why a model reached a particular conclusion or whether that conclusion is appropriate for a patient. This...
U. Mehta· International Journal of Cre...· 0 citations
A review examines how free full-text PubMed literature describes the real-world implementation of AI in healthcare and identifies the technical, organizational, ethical, and social conditions that shape adoption.
Jagoda Pałubska, Oliwer Műller, Zuzanna Rafałowska et al.· International Journal of Inn...· 0 citations
The protocol emphasizes several key components, including the preparation of standardized healthcare datasets, the development of stable machine learning models, the evaluation of predictive performance, the generation of clinically relevant explanations, and the statistical assessment of reproducibility across repeate...
Sital Dash, Kailas Patil· Journal of Visualized Experi...· 1 citation
This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.
S. E. McKinney, Phuc Vu, Samuel A. Justice et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.