Mitigating Sycophancy and Alignment Failures in Large Language Models

Abstract

Large language models (LLMs) that have been fine-tuned using Reinforcement Learning from Human Feedback (RLHF) are likely to conform to the user, even when the user is wrong. This is one of the more intransigent side effects of current alignment methods and is called sycophancy. This review traces the origin of sycophancy and what has been attempted in the quest to correct it. We explain the RLHF and Direct Preference Optimization (DPO) pipelines, and how reward model overoptimization, dynamics of Goodhart’s Law, and biased preference data are all integrated to yield agreeable-but-wrong output. We categorize the behavior into six subtypes and organize the work on mitigation methods into four categories: training-time methods (DPO, Constitutional AI, RLAIF), inference-time methods (activation steering, representation engineering), reward model hardening, and evaluation benchmarks. The algorithms evaluated are PPO-RLHF, DPO, IPO, KTO, and CAI. We round off by talking about what is not yet figured out: the balance between helpfulness and honesty, whether these solutions are likely to be effective at frontier scale, and that most of the alignment work has been conducted in the context of English and Western worlds.