OpenFace-Adaptive: Robust Multimodal Emotion Recognition via Reliability-Aware Gating and Heterogeneous Graph Attention

Abstract

Multimodal emotion recognition systems typically assume complete, noise-free data from all sensors. However, in "in-the-wild" deployments, modalities such as face video are frequently occluded or corrupted, causing existing fusion methods to degrade catastrophically. We propose OpenFace-Adaptive, a novel framework featuring a Self-Supervised Reliability Gating mechanism that assigns dynamic trust scores to each modality via reconstruction autoencoders, selectively dampening noisy signals before fusion. We further employ a Heterogeneous Graph Attention Network (HGAT) with distinct edge-type projection matrices to model diverse cross-modal dependencies, alongside a Cross-Modal InfoNCE Loss for contrastive representation alignment. Experiments on the CMU-MOSEI benchmark with rigorous 5-seed evaluation and statistical significance testing demonstrate that our gated model maintains stable accuracy under diverse sensor corruptions—including modality dropout, Gaussian noise, partial occlusion, and feature shuffling—outperforming ungated baselines by up to 3.1 percentage points under audio corruption. The model achieves 81.5% binary accuracy on clean data and a compact footprint of 1.9MB after INT8 quantization, demonstrating potential for Edge AI deployment.