Multimodal Fusion of Electrocardiogram Signals and Demographic Characteristics with Cross-Attention Networks in the Classification of Cardiovascular Diseases

Abstract

Cardiovascular diseases rank first among the causes of mortality worldwide. In the evaluation of these diseases, electrocardiography (ECG), which provides low-cost and non-invasive electrical mapping, is used as the primary tool for diagnosis. The manual examination of ECG recordings obtained from a wide range of patients creates time cost and workload for specialists. Therefore, in order to increase accuracy and support specialists, there are many deep learning-based ECG interpretation approaches in the literature. In this study, it is aimed to simultaneously classify five main classes of cardiovascular diseases (NORM, MI, STTC, CD, HYP) using data obtained from the PTB-XL database and defined with SCP diagnostic codes. The proposed architecture in the study consists of three main components. As the first component of the proposed architecture, second-degree polynomial cross-feature transformation was applied to gender, age, height, and weight variables, and these data were transformed into a high-dimensional representation space with dmodel dimension through a neural network containing the GELU (Gaussian Error Linear Unit) activation function. As the second component, after separating 12-channel ECG recordings into multi-scale local representations through parallel convolution operations (1D-CNN), the temporal dependencies of the signals were modeled with xLSTM (mLSTM) layers. With the cross-attention module, which is the third component, the clinical data obtained from the first component were taken as ‘query’ and the ECG data obtained from the second component were taken as ‘key/value’; after being fused, predictions for five disease classes were generated through the obtained combined five independent context vectors. The model training process was designed to prevent information leakage and was carried out with BCEWithLogitsLoss compatible with multi-label classification. The learning rate and AdamW weight decay selections were optimized with a 12-cell grid sweep in order to maximize the potential of the model, and each combination was improved with 5 independent seeds. In the study, ablation studies were conducted to evaluate the effects of the data coming from two components rather than relying on a single model performance. In addition, the proposed model was compared with models such as ResNet and InceptionTime, and statistically summarized within a 95% confidence interval over five different seeds. As shown in Table 6, the macro AUROC score of 0.9106 and the macro F1 score of 0.7151 represent the average values obtained from five independent seed values, indicating that the proposed method exhibits successful performance. In addition, the heat maps obtained by overlaying the attention weights on the ECG signal provide temporal interpretability to the model’s decision-making mechanism. As a result, it presents a methodological framework consistent with the literature, supported by multimodal data fusion, extensive statistical analyses, and interpretability steps.