Is Sycophancy a Form of In-Context Updating?
TL;DR
It is proposed that factual sycophancy arises when in-context updating gives a user's assertion excessive weight relative to its evidence, a trade-off that calls for evaluations measuring both behaviors and mitigations that decouple them.
Abstract
Factual sycophancy, a language model's deference to a user's false beliefs, is usually understood as a bid for the user's approval at the expense of truthfulness. We argue that this framing over-emphasizes the user and propose viewing sycophancy as a failure within the broader and generally useful process of in-context updating, by which models revise their answers based on information supplied during an interaction. We support this view with behavioral and mechanistic evidence. First, models'responses to user claims sit within a broader pattern of source-sensitive updating. They revise their answers in response to information attributed to external sources as well, more so when it appears better supported, and their deference to the user persists even when no user approval is at stake. Second, the internal components that support deference to the user are also causally important for in-context updating more broadly. Our results suggest that factual sycophancy arises when in-context updating gives a user's assertion excessive weight relative to its evidence. Because sycophancy is embedded in this process, reducing it risks reducing useful updating as well, a trade-off that calls for evaluations measuring both behaviors and mitigations that decouple them.
