Multimodal deep learning outperforms clinical and brain region models in predicting stroke-associated pneumonia: an explainable AI study

Abstract

Background Stroke-associated pneumonia (SAP) is a frequent complication after acute ischemic stroke (AIS) and is associated with poor outcomes. This study aimed to develop an interpretable multimodal deep learning model integrating MRI, lesion-related brain regions, and clinical variables for early SAP prediction. Methods A total of 426 AIS patients were retrospectively enrolled, including 71 patients with SAP. Multimodal MRI data (DWI, T1WI, and T2-FLAIR) were processed using standardized registration and lesion segmentation. A 3D convolutional neural network was used to extract imaging representations, which were fused with clinical variables and AAL3-based brain-region features. Model performance was assessed using stratified five-fold cross-validation and nested cross-validation when applicable, with further evaluation based on receiver operating characteristic (ROC) analysis, calibration analysis, and decision curve analysis. Grad-CAM was applied for model interpretation. Results The multimodal fusion model achieved the best performance for SAP prediction, with an AUC of 0.782 (95% CI: 0.712–0.839), compared with the clinical model based on conventional clinical variables (AUC = 0.756), the imaging model based on 3D CNN representations (AUC = 0.693), and the brain-region model based on AAL3-derived lesion location features (AUC = 0.501). The fusion model showed superior clinical utility and favorable calibration. Grad-CAM visualization demonstrated that model predictions were mainly driven by lesion-related cortical and subcortical regions. Conclusion A multimodal deep learning framework integrating MRI, brain-region information, and clinical characteristics improved SAP prediction after AIS and provided an interpretable approach for individualized risk stratification.