Preprint

From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

Zhibo Hou Fan Zhao Zhi-Yu An Wan Du

Aug 2026 · 0 citations · 41 references

Computer Science

TL;DR

This work proposes Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection that substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.

Abstract

Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.

arXiv Semantic Scholar

View source

Similar papers

Jul 2026

AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning

AdaKP is an online selector that re-chooses each problem's KP subset over the course of RL training, an entropy proxy that scores a KP by the reduction in next-token entropy it induces in a single inexpensive forward pass, with a provable bound on its truncation bias.

Zibin Meng, Zhenyu Zhao, Chunqiang Run · arXiv.org · 0 citations

#machine learning Jul 2026

Distilled Reinforcement Learning for LLM Post-training

Extensive experiments show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k, and can effectively transfer previously unavailable knowledge from a teacher model to a student model.

Chen Wang, Zhaochun Li, Jionghao Bai et al. · arXiv.org · 2 citations

Preprint Aug 2026

RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection

RING (Retrieval-Internalized Generation), a holistic paradigm spanning both architecture and training that injects large-scale external knowledge into aixture-of-Memory Experts and learns parametric search over this internal memory via reinforcement learning, removing the external retriever entirely, is proposed.

Shicheng Xu, Liang Pang, Li-Yi Chen et al. · 1 citation

#machine learning Preprint Sep 2026

Cliff: Learning Process Rewards from the First Mistake

Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout, is proposed and established as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Pei-Xuan Han, Runnan Wang, Ketan Ramaneti et al. · 1 citation

Preprint Jul 2026

Cross-Benchmark Generalization in Long-Horizon Agents

Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.

Sushant Mehta, Logan Ritchie, Liudas Panavas et al. · 0 citations

Preprint Aug 2026

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Self-Reflective Policy Optimization (SRPO) enables LLMs to analyze their own completed trajectories, synthesize errors into concise"reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals.

Jialong Liu, Yu-Ling Shi, Ning Yang et al. · 2 citations