Preprint

Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy

Jim Dilkes V. Yazdanpanah Sebastian Stein

Aug 2026 · 0 citations · 44 references

Computer Science

TL;DR

Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted, improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks.

Abstract

Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.

arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

Cross-Benchmark Generalization in Long-Horizon Agents

Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.

Sushant Mehta, Logan Ritchie, Liudas Panavas et al. · 0 citations

#machine learning Preprint Aug 2026

Demystifying Reinforcement Learning Post-Training of Language Models

This work deconstructs the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface, and uses the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shape...

D. Clay, Saket Gollapudi, Sankar V Harilal et al. · 1 citation

Jul 2026

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model, is proposed.

Jinyang Wu, Shuo Yang, Zhengxi Lu et al. · arXiv.org · 7 citations

#machine learning Preprint Sep 2026

Cliff: Learning Process Rewards from the First Mistake

Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout, is proposed and established as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Pei-Xuan Han, Runnan Wang, Ketan Ramaneti et al. · 1 citation

Preprint Aug 2026

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

This work proposes BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively, enabling agents to utilize external skills more effectively.

Tian Pan, Yuan Li, Hong-Da Wang et al. · 1 citation

Review Aug 2026

Agentic Reinforcement Learning with Self-Distilled Reward Shaping

ADRS is introduced, a framework for constructing return-associated token-level credit for multi-turn language agents that centers and normalizes privileged token scores within each step, modulates them with a return-associated Teacher Value Advantage gate based on within-group confidence--return association, and incorp...

Ran Zhang, Guinan Chen, Chenshaodong et al. · 1 citation