Measuring and Mitigating (Over)reliance to Build Human-Compatible AI

Abstract

Large language models (LLMs) differ from previous technologies in functioning as collaborative "thought partners'' that can engage fluidly in natural language across a range of tasks and increasingly act as autonomous agents on users' behalf. As they come to shape consequential decisions from healthcare to personal advice, understanding how users rely on them, and when that reliance becomes miscalibrated, grows more urgent. This paper argues that measuring and mitigating overreliance must become central to LLM research and deployment. First, we consolidate risks from overreliance at both the individual and societal levels, including high-stakes errors, governance challenges, and long-term impacts on skill development and autonomy. Then, we explore LLM characteristics, system design features, and user cognitive biases that together raise serious and unique concerns about overreliance on LLMs in practice. We also examine historical approaches for measuring reliance, identifying three gaps and proposing three directions to improve measurement. Finally, we discuss mitigation strategies that can be pursued to ensure LLMs augment rather than diminish human capabilities.