Deep Learning-Based Multimodal Fusion for Early Prediction of Cardiovascular Risk Using Clinical and Imaging Data
TL;DR
This work proposes a multimodal deep fusion framework with attention for high accurate cardiovascular risk stratification using the integration of medical images and clinical data and demonstrates that this adaptive fusion strategy outperforms simple concatenation baselines.
Abstract
About this, cardiovascular disease is the leading cause of death worldwide, and risk prediction nowadays can be made using only clinical data or imaging data alone, missing the opportunity to leverage the complementary information that could be acquired from combining both sorts of data. To this end, we propose a multimodal deep fusion framework with attention for high accurate cardiovascular risk stratification using the integration of medical images and clinical data. The system works in a parallel pipeline in which clinical features are extracted using a fully connected neural network (with Batch normalisation and Dropout) and imaging features are extracted using a fine-tuned CNN or Vision Transformer encoder. The most striking innovation is the presence of this “learnable” attention module, which learns the modality-specific scalar weights for each modality at each time step so as to adaptively emphasise the most-predictive signals for a particular patient.This approach mitigates the influence of noisy or less informative data sources and produces a fused representation that is subsequently passed through a deep classification head. The entire architecture is trained end-to-end using the Adam optimizer with binary cross-entropy loss. We demonstrate that this adaptive fusion strategy outperforms simple concatenation baselines by capturing complex interactions between systemic physiological states and morphological patterns, such as ventricular geometry and vascular irregularities. Furthermore, the framework is designed to handle both binary and multi-class risk assessment tasks.