Robust multimodal emotion recognition under incomplete modalities via transformer conditional variational autoencoders
TL;DR
RMM-TCVAE, a robust MER framework based on a Transformer-based Conditional Variational Autoencoder, combines modality-specific encoders, a conditional Transformer Reconstruction Network for recovering missing modalities, and a Gated Multimodal Fusion Network for adaptive cross-modal integration.
Abstract
Summary Multimodal emotion recognition (MER) has gained increasing research interest owing to its ability to integrate complementary emotional cues from textual, acoustic, and visual modalities. However, real-world applications often suffer from incomplete multimodal inputs caused by sensor failures, communication disruptions, or noisy environments, resulting in missing modalities or corrupted features. To address these challenges, this study proposes RMM-TCVAE, a robust MER framework based on a Transformer-based Conditional Variational Autoencoder. The framework combines modality-specific encoders, a conditional Transformer Reconstruction Network for recovering missing modalities, and a Gated Multimodal Fusion Network for adaptive cross-modal integration. Experiments on CMU-MOSI, CMU-MOSEI, and IEMOCAP demonstrate that RMM-TCVAE consistently outperforms strong baselines under incomplete multimodal conditions, achieving 84.8% accuracy on CMU-MOSEI and yielding improvements of up to 2.9 percentage points in Acc and 3.1 percentage points in WAF across fixed missing-rate experiments. Ablation and visualization analyses support the effectiveness of RMM-TCVAE under incomplete multimodal conditions.