Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alte...
Oliver Daniels, Perusha Moodley, Benjamin M. Marlin et al.
· 0 citations