Dual-Group Exploration: Reinforcement Learning for Video Understanding in Multimodal Large Language Models

Abstract

Reinforcement Learning (RL) has significantly enhanced reasoning in Large Language Models. However, applying state-of-the-art RL methods like Group Relative Policy Optimization (GRPO) to video-based Multimodal LLMs (MLLMs) remains sub-optimal. We identify two key exploration bottlenecks: (1) Advantage Vanishing : generating multiple responses from invariant video contexts leads to near-uniform rewards, neutralizing learning signals after in-group standardization; (2) Visual Over-fitting : standard RL lacks mechanisms to regularize sensitivity to visual perturbations under single-epoch training, causing policies to over-fit to sparse temporal cues. To address these, we propose Dual-Group Exploration, a framework revitalizing learning signals at both input and output levels. At the input level , a Dual-Group Sampling strategy breaks visual determinism via complementary temporal sequences. At the output level , an Entropy-rated ELO Advantage captures inter-group quality variances. Concur-rently, a Cross-Visual Counterfactual Weight establishes a Visual Consistency Region, safely driving exploration by enforcing reasoning consistency across varied views. Achieving state-of-the-art performance across six video reasoning benchmarks, our method demonstrates that consistency-guided exploration is beneficial for video understanding.