CLMER: A Framework for Contrastive Learning-Based Multimodal Emotion Recognition.

Abstract

Emotion recognition plays a crucial role in human-computer interaction and affective computing, yet its effectiveness is limited by the difficulty of integrating heterogeneous modalities with fundamentally different structures, such as physiological signals and visual data. In this article, we propose CLMER, a contrastive learning-based multimodal cross-attention framework designed to address the challenges of complex emotion recognition. The framework introduces a serialization strategy that converts pixel-level image data into time-series data, aligning it with the temporal characteristics of physiological signals. CLMER consists of three core components that work together to enable effective multimodal emotion recognition. The multimodal data preparation module preprocesses physiological and visual data, ensuring consistency across modalities. Building on this foundation, the contrastive learning (CL)-based feature extraction module generates temporal representations that capture the essential patterns embedded in the data through self-supervised (SS) learning. Finally, the multimodal fusion module employs cross-modal attention to integrate features with improved modality alignment. Experimental evaluations on two public datasets DEAP, AMIGOS and a private dataset MAN-II demonstrate that CLMER significantly outperforms unimodal and traditional fusion approaches, achieving state-of-the-art performance in emotion classification tasks. These findings highlight the framework's robust generalization, computational efficiency, and strong performance in multimodal emotion recognition, suggesting its potential for real-world deployment. Our code is available at https://github.com/liangyubuaa/CLMER.