An Attention-Based TabTransformer Framework for Heart Disease Prediction

Abstract

Cardiovascular diseases remain top in the world mortality statistics even after significant progress in diagnosing and treating the disease. The recognition of the people at risk prior to the emergence of the critical symptoms is thus an important measure to the minimization of the mortality levels and prevention of the secondary outcomes. Clinical data gathered through the electronic health systems have been applied in a wide range of machine learning techniques to this problem. Classical algorithms such as those based on logistic regression, decision trees, K-nearest neighbours, Naive Bayes, and support vector machines have reported encouraging yet limited performance largely because they are shallow learners and because they rely on engineered features. Advanced algorithms like the Random Forests, XGBoost, and other ensemble learners have been making significant strides but even at the cost of working on independent extracted attributes, with deep learning systems slowly replacing these older model systems. Nonetheless, the widely used architectures, which include multilayer perceptrons, CNNs and hybrid CNN-LSTM based models, are based on spatial or temporal structure built in the data, which is absent in the tabular clinical data. To manage this mismatch, this study presents an attention-based TabTransformer framework, which considers categorical attributes as learnable tokens and captures the dynamics of attributes among them with the multi-head self-attention. Parallel dense projections are used to include numerical attributes and the result is merged with the contextualized token representations to do the final prediction.