An Explainable Hybrid Multimodal Learning Framework for Heart Disease Prediction Using Clinical and Imaging Data

Abstract

The increasingly sophisticated nature of modern-day healthcare requires intelligent solutions that will help merge several types of datasets and produce more accurate disease prediction results. Conventional approaches that employ only either structured medical data or images ignore numerous essential factors that contribute to better understanding of the patient’s condition. Therefore, this paper offers a new solution that allows integrating multiple technologies into the process of disease prediction. Machine learning, deep learning, as well as explainable AI, are incorporated into the system, which includes XGBoost model that analyzes structured data from the UCI Heart Disease dataset and an EfficientNetB0 neural network designed to classify skin lesions with the help of the HAM10000 dataset. An additional layer of explainability is provided by SHAP analysis of the clinical model and the utilization of Grad-CAM in order to visualize imaging results. Moreover, a simple technique that allows integrating results obtained with both approaches in one decision-making process is discussed. Clinical model achieves 88.52% accuracy and AUC equals 0.94, whereas the accuracy of the imaging model makes 76.88%.It ensures that decisions are made based on sound judgment through the utilization of the data from both approaches. It is important to note that although the data from both models is not linked to the same patients, the model demonstrates how AI multi-models can easily scale and be explained.