Skip to content
Preprint

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

Oct 2026 · 1 citation · 25 references
Computer Science

TL;DR

This work introduces MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD), and the Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions.

Abstract

Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.

View source

Similar papers

Preprint Oct 2026

Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents

Long-MDR is introduced, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue that improves both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct trai...

On-Tai Tang, Huan-Yao Zhang, Jie-Peng Zhou et al. · 0 citations
#machine learning Preprint Oct 2026

TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery

Emerging long-horizon agentic tasks require repeated model calls, worsening the inference cost of already-costly language models. While narrow agentic tasks suggest potential for aggressive model pruning without performance drop, empirical results show existing methods proposed for question answering tasks severely deg...

Yuan-Zhe Li, Peng-Xin Wang, Yuxin Ren et al. · 0 citations
#artificial intelligence Preprint Oct 2026

LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

Deep search agents tackle complex knowledge tasks through iterative retrieval, multi-hop reasoning, and evidence synthesis across multiple sources. Existing approaches typically assume relatively stable retrieval systems and operate over short-horizon tool interaction. However, when retrieval is sensitive to query form...

Zhi-Qing Nong, Liang Wen, Chao-Hsuan Liu · 0 citations
#artificial intelligence Preprint Oct 2026

Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards

Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments. Self-play offers an alternative, but must generate achievable, compositional tasks and provide feedback beyond bin...

Tian-Yu Liu, Davide Belli, Bence Major et al. · 0 citations
#artificial intelligence Conference Open access Sep 2026

IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesis

Deep search requires LLM agents to decompose complex queries, search for evidence, and synthesize grounded answers, yet existing ReAct-style agents suffer from two limitations: role coupling, where one policy must handle planning, evidence use, and synthesis; and context accumulation, where growing search histories int...

Xing-Yu Wu, Yuchen Yan, Zhengxi Lu et al. · 0 citations
Preprint Oct 2026

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better s...

Ying-Ping Wang, Min Cai, Chen-Yu Yuan et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.