Multimodal Active Learning via Emotion-Aware Cross-Modal Contrastive Learning

Abstract

Multimodal Emotion Recognition (MER) leverages diverse modalities such as text, vision, and audio to infer emotional states, yet suffers from prohibitively high annotation costs. As a promising solution, Active Learning (AL) strategically selects the most valuable data for annotation, enabling competitive model performance with substantially reduced labeling effort. To address this issue, this study proposes MALCC, a novel multimodal active learning framework that leverages emotion-aware cross-modal contrastive learning for emotion recognition. Specifically, to achieve task-specific alignment across modalities, we first design an emotion-aware cross-modal contrastive learning module, which embeds multiple modalities into a unified emotion-aligned representation space. Within this space, we propose a comprehensive reliability-aware query strategy that prioritizes cross-modal semantic consistency and modality contribution balance as two key reliability criteria, while incorporating informativeness and representativeness as complementary utility criteria to select more valuable instances for annotation. By leveraging this strategy, we then design a novel multimodal active learning algorithm tailored for emotion recognition. We further provide theoretical analysis to justify our design and conduct extensive experiments on three widely used MER datasets, demonstrating superior performance over state-of-the-art (SOTA) methods. Our code and supplementary materials are available at https://github.com/Atlantis-esh/malcc0425.