Aditya Agrawal

Can You Trust Your Preference Models? Stackelberg Games for Alignment

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are un...