Long-MDR is introduced, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue that improves both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly.
Abstract
The next generation of multimodal research agents must reason over long-lived research histories rather than short model completions. During a single task, an agent may repeatedly search the web, inspect visual evidence, revisit earlier hypotheses, and accumulate tens of thousands of tokens of multimodal context. Despite this trend, online RL for multimodal research agents remains largely confined to shorter contexts and interaction horizons. We push online RL training to 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context, and the first trained with a 75 tool-turn horizon. Scaling to this regime exposes several practical limitations of conventional RL training. Early in training, weak policies make poor use of large interaction budgets, causing expensive rollouts with little reward improvement. Later, policy entropy can collapse before performance has saturated, prematurely ending useful learning. We introduce Long-MDR, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques improve both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly. Code will be available at https://github.com/navileon/Long-MDR
This work introduces MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD), and the Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions.
Huan-Yao Zhang, Jie-Peng Zhou, Jia-Ye Lin et al.· 1 citation
OP-CL is proposed, a two-stage on-policy guided framework for continual post-training in MLLMs that achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.
Xing-Yue Guo, Jia-He Wu, Xu-Hui Sui et al.· Proceedings of the 34th ACM...· 0 citations
: Deep reinforcement learning (DRL) has made significant progress in recent years. However, it still faces multiple bottlenecks in real-world applications. Agents suffer from low sample efficiency, and designing effective reward functions remains very difficult. Furthermore, traditional DRL lacks general cognitive abil...
Sheng-Hao Yuan· Proceedings of the 4th Inter...· 0 citations
This work introduces TrajLong, a novel framework that compiles trajectories into long-context training tasks with dense supervision, targeting three representative atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance, and investigates the relationship between long-context...
Miao Peng, Qin-Tong Zhang, Nuo Chen et al.· 0 citations
RewardWeaver is introduced, a self-evolving reward adaptation framework for language agents in long-horizon interaction that maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward a...
Heng-Bo Xiao, Bo-Yao Zhang, Pu-Rui Liu et al.· 0 citations
This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better s...
Ying-Ping Wang, Min Cai, Chen-Yu Yuan et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.