Skip to content
Preprint

Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents

Oct 2026 · 0 citations · 40 references
Computer Science

TL;DR

Long-MDR is introduced, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue that improves both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly.

Abstract

The next generation of multimodal research agents must reason over long-lived research histories rather than short model completions. During a single task, an agent may repeatedly search the web, inspect visual evidence, revisit earlier hypotheses, and accumulate tens of thousands of tokens of multimodal context. Despite this trend, online RL for multimodal research agents remains largely confined to shorter contexts and interaction horizons. We push online RL training to 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context, and the first trained with a 75 tool-turn horizon. Scaling to this regime exposes several practical limitations of conventional RL training. Early in training, weak policies make poor use of large interaction budgets, causing expensive rollouts with little reward improvement. Later, policy entropy can collapse before performance has saturated, prematurely ending useful learning. We introduce Long-MDR, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques improve both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly. Code will be available at https://github.com/navileon/Long-MDR

View source

Similar papers

Preprint Oct 2026

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

This work introduces MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD), and the Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions.

Huan-Yao Zhang, Jie-Peng Zhou, Jia-Ye Lin et al. · 1 citation
Book Open access Nov 2026

OP-CL: On-Policy Guided Continual Post-training for Multimodal Large Language Models

OP-CL is proposed, a two-stage on-policy guided framework for continual post-training in MLLMs that achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.

Xing-Yue Guo, Jia-He Wu, Xu-Hui Sui et al. · 0 citations
Conference Open access 2026

Towards Automated Reinforcement Learning: Applications and Prospects of Large Language Models as Cognitive Components in the Full RL Lifecycle

: Deep reinforcement learning (DRL) has made significant progress in recent years. However, it still faces multiple bottlenecks in real-world applications. Agents suffer from low sample efficiency, and designing effective reward functions remains very difficult. Furthermore, traditional DRL lacks general cognitive abil...

Sheng-Hao Yuan · 0 citations
#natural language process... Preprint Oct 2026

TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training

This work introduces TrajLong, a novel framework that compiles trajectories into long-context training tasks with dense supervision, targeting three representative atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance, and investigates the relationship between long-context...

Miao Peng, Qin-Tong Zhang, Nuo Chen et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

RewardWeaver is introduced, a self-evolving reward adaptation framework for language agents in long-horizon interaction that maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward a...

Heng-Bo Xiao, Bo-Yao Zhang, Pu-Rui Liu et al. · 0 citations
Preprint Oct 2026

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better s...

Ying-Ping Wang, Min Cai, Chen-Yu Yuan et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.