Skip to content
Book Open access

OP-CL: On-Policy Guided Continual Post-training for Multimodal Large Language Models

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 2644-2653 · 0 citations · 4 references

TL;DR

OP-CL is proposed, a two-stage on-policy guided framework for continual post-training in MLLMs that achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.

Abstract

Continual post-training is essential for adapting Multimodal Large Language Models (MLLMs) to evolving tasks, but it often leads to catastrophic forgetting. The core challenge is to achieve rapid adaptation to new task formats and semantics while preventing policy drift from accumulating into forgetting during sequential training. Existing paradigms address this trade-off only partially: Supervised Fine-Tuning (SFT) enables rapid adaptation but often disrupts prior knowledge, whereas Reinforcement Learning (RL) better preserves existing capabilities through on-policy updates but typically converges slowly and struggles on open-ended tasks. We propose OP-CL, a two-stage on-policy guided framework for continual post-training in MLLMs. In Stage 1, On-Policy Corrective Supervision transforms model rollouts into corrective supervision, providing efficient learning signals for acquiring new-task behaviors, especially on open-ended tasks. Since such rapid adaptation may still introduce policy deviation, Stage 2, Reward-KL Stabilized Reinforcement further stabilizes continual adaptation by adaptively regulating KL regularization to limit excessive policy updates. The two stages therefore address two complementary requirements of continual post-training: efficient acquisition and stable retention. Experiments show that OP-CL consistently achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.

Read PDF

Similar papers

#artificial intelligence Open access Oct 2026

TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training

Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient bu...

Yu-Qian Fu, Kai-Wen Jiang, Zi-Jun Song et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In prac...

C. Wu, Thomas T. C. K. Zhang, Aditi Raghunathan · 1 citation
Preprint Oct 2026

Continual Test-Time Learning with Evolution Strategies

Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward sign...

Guang-Ya Hao, Yi-Ting Duan, Long-Yan Tan et al. · 0 citations
Preprint Nov 2026

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

Continual pre-training (CPT) adapts large language models (LLMs) to specialized domains, yet its standard next-token prediction (NTP) objective treats every corpus token as a fixed target, regardless of its utility to the current model. In a controlled study on the same 1B-token mathematical corpus, we observe sharply...

Rui-Ze Zheng, Xu-Yang Liu, Siyu Ren et al. · 0 citations
Preprint Oct 2026

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better s...

Ying-Ping Wang, Min Cai, Chen-Yu Yuan et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.