Hierarchical Quantized Cross-modal Masked Autoencoders for Audio-Visual Emotion Recognition

Abstract

Cross-modal self-supervised learning has shown strong potential for learning representations for audio-visual emotion recognition (AVER). However, most existing methods operate entirely in continuous embedding spaces, which can inadvertently preserve irrelevant patterns, spurious correlations, and modality-specific noise. To improve robustness and generalization of AVER models under noisy conditions and domain shift, we propose the hierarchical quantized cross-modal masked autoencoders (HQC-MAE) framework. This method integrates discrete representation learning into masked data modeling and contrastive learning via learned codebooks, creating hierarchical information bottlenecks in a multimodal setting. We perform cross-modal contrastive learning in a discrete space, encouraging semantically aligned audio and video samples to match in a structured embedding space. Our vector quantization modules are integrated into pretrained unimodal foundation models (VideoMAEv2 for video and AudioMAE++ for audio), enabling us to leverage large-scale pretrained unimodal models rather than training them from scratch. Experiments on emotion recognition benchmarks across diverse settings show that HQC-MAE consistently outperforms AVER baselines using continuous representations.