Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation
This paper observes that when a safe model is recursively self-distilled on its own (safe) generations while receiving an additional non-harmful cue, it becomes increasingly misaligned across a diverse set of harmful queries; it is called Progressive Misalignment.
Karim Abdel Sadek, F. Di Gennaro, Fanny Yang
· 0 citations