Nov 2026· Proceedings of the 34th ACM International Conference on Multimedia· pp. 2644-2653· 0 citations· 4 references
TL;DR
OP-CL is proposed, a two-stage on-policy guided framework for continual post-training in MLLMs that achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.
Abstract
Continual post-training is essential for adapting Multimodal Large Language Models (MLLMs) to evolving tasks, but it often leads to catastrophic forgetting. The core challenge is to achieve rapid adaptation to new task formats and semantics while preventing policy drift from accumulating into forgetting during sequential training. Existing paradigms address this trade-off only partially: Supervised Fine-Tuning (SFT) enables rapid adaptation but often disrupts prior knowledge, whereas Reinforcement Learning (RL) better preserves existing capabilities through on-policy updates but typically converges slowly and struggles on open-ended tasks. We propose OP-CL, a two-stage on-policy guided framework for continual post-training in MLLMs. In Stage 1, On-Policy Corrective Supervision transforms model rollouts into corrective supervision, providing efficient learning signals for acquiring new-task behaviors, especially on open-ended tasks. Since such rapid adaptation may still introduce policy deviation, Stage 2, Reward-KL Stabilized Reinforcement further stabilizes continual adaptation by adaptively regulating KL regularization to limit excessive policy updates. The two stages therefore address two complementary requirements of continual post-training: efficient acquisition and stable retention. Experiments show that OP-CL consistently achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.
Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient bu...
Yu-Qian Fu, Kai-Wen Jiang, Zi-Jun Song et al.· Conference on Empirical Meth...· 0 citations
A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In prac...
C. Wu, Thomas T. C. K. Zhang, Aditi Raghunathan· 1 citation
Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward sign...
Guang-Ya Hao, Yi-Ting Duan, Long-Yan Tan et al.· 0 citations
Continual pre-training (CPT) adapts large language models (LLMs) to specialized domains, yet its standard next-token prediction (NTP) objective treats every corpus token as a fixed target, regardless of its utility to the current model. In a controlled study on the same 1B-token mathematical corpus, we observe sharply...
Rui-Ze Zheng, Xu-Yang Liu, Siyu Ren et al.· 0 citations
This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better s...
Ying-Ping Wang, Min Cai, Chen-Yu Yuan et al.· 0 citations
This work introduces ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations.