A Unified Explainable AI Framework for Multimodal Chest X-Ray and Clinical Note Diagnosis

Abstract

Chest radiography is the most common first-line imaging modality for diagnosing respiratory disease, but deep learning classifiers built for this task are largely opaque, which restricts their clinical adoption. This paper proposes a multimodal explainable-AI (XAI) framework that couples a DenseNet-121 convolutional network for chest X-ray classification with a RadBERT transformer for clinical-note analysis inside a single interpretability pipeline. Grad-CAM produces spatial saliency maps for the imaging branch, while SHAP-based token attribution explains the predictions of the language branch, whose pathology label is obtained through cosine similarity against class-prototype embeddings. On a representative case, the text branch assigned an 80.8% similarity score to pneumonia and highlighted clinical indicators such as fever and consolidation alongside demographic terms such as age and diabetes history. Rather than reporting a marginal accuracy improvement, the central contribution is a cross-modal consistency check that compares the imaging and textual explanations for the same patient and automatically surfaces discrepancies — for example, a mismatch between the anatomical location implicated by the image and the one described in the note — that neither modality reveals in isolation. Outputs are exposed through a standardized JSON schema together with a natural-language summary to support clinician review.