VGMR: Cross-Modal Value-Gated Modality Refiner for Robust Multimodal Learning
Abstract
Multimodal learning combines heterogeneous signals, such as text, audio, and vision, to improve prediction. However, adding modalities does not always help, because modalities may contain useful complementary cues together with noisy, redundant, or contextually unreliable components that interfere with full-modality learning. In this work, we propose Cross-Modal Value-Gated Modality Refiner (VGMR), a pre-fusion refinement module that estimates context-conditioned value evidence before modality features enter the backbone. VGMR models summary-level cross-modal agreement and discrepancy, estimates global-level and channel-level value signals, and uses them with raw and projected features to generate fine-grained gates over the original modality features. Different from conventional attention or generic gating, VGMR treats value as signed, context-conditioned impact evidence rather than a direct preservation probability, allowing the gate generator to learn whether each component should be enhanced, retained, or suppressed. We evaluate VGMR on MOSI and MOSEI with a Transformer backbone, and on MOSI, MOSEI, UCF101, AVE, and CREMA-D with a ResNet+Concat backbone. On MOSEI with the Transformer backbone, VGMR achieves 0.8446 Accuracy and 0.8029 Macro-F1, outperforming the strongest baselines by 1.09 and 0.89 percentage points, respectively. Ablation, controlled comparison, modality removal, noise robustness, optimization, and qualitative analyses suggest that VGMR can reduce unreliable modality responses before fusion and improve robustness under noisy multimodal inputs.