Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Design
On-policy RL preserves a safety buffer inherent in the model's own generation distribution, one that is bypassed during off-policy settings, one that is bypassed during off-policy settings.