Multimodal graph neural networks for integrating heterogeneous medical data in clinical decision making

Abstract

Healthcare data is complex, with Electronic Health Records (EHRs), clinical images, notes, and genetic data, challenging and enabling the development of effective clinical decision support systems. In this paper, we propose a multi-modal Graph Neural Network (GNN) model, DUA-HIR, which combines structured and unstructured information, such as text and images, to enhance diagnosis and risk prediction. It leverages GNNs to model patient relationships, Convolutional Neural Networks (CNNs) to process images, and Transformers to analyse text. It uses attention and multi-task learning for improved predictions of clinical outcomes. The model is thoroughly evaluated on MIMIC-III and MIMIC-CXR, showing substantial improvements in a range of key clinical prediction tasks. Ablation experiments, sensitivity analyses, calibration measures and computational costs confirm the model’s architecture and practicality. The paper also explores ethical implications for clinical use.