Dual-contrastive modality recovery for incomplete multi-modal brain disease diagnosis

Abstract

Multi-modal learning is extensively applied to diagnose brain diseases such as epilepsy and Alzheimer's disease. However, incomplete multi-modal data, where some imaging modalities are unavailable or difficult to collect, limits the application of conventional methods. Additionally, existing approaches primarily focus on reconstructing missing imaging data but rarely enforce cross-modal semantic alignment. To address these challenges, we propose BrainCLIP, a CLIP inspired two-stage framework designed for incomplete multi-modal learning, with a focus on diagnosing representative brain diseases, i.e., epilepsy and Alzheimer's disease. The key novelty of our framework lies in its joint design of dual contrastive modality recovery and multi-modal representation learning in a two-stage pipeline. Specifically, we introduce a multi-modal contrastive learning stage that aligns text, fMRI, and DTI representations in a shared embedding space using complete samples. The recovered features are then refined through a dual contrastive recovery strategy with modality level and sample level contrastive objectives, thereby ensuring that the recovered features are semantically consistent and discriminative. For multi-modal representation learning, the recovered and available modalities are fused with fixed textual embeddings to learn task aware representations for disease classification. Extensive experiments demonstrate the effectiveness of our method in diagnosing epilepsy and Alzheimer's disease.