Intent-Description Anchoring Bias in LLM-as-a-Judge Evaluation of Recommender Systems
TL;DR
It is found that LLM judges are influenced by descriptions of a recommendation algorithm’s optimization objective, even when evaluating identical recommendation outputs, a phenomenon the authors term intent-description anchoring bias.
Abstract
Large Language Models (LLMs) are increasingly used to evaluate recommendation systems, but are known to exhibit systematic biases. We study whether LLM judges are influenced by descriptions of a recommendation algorithm’s optimization objective, even when evaluating identical recommendation outputs. We find that they are, a phenomenon we term intent-description anchoring bias, where an algorithm’s stated objective influences judgments beyond what is supported by the recommendations themselves. Across four frontier LLMs from three commercial providers, providing algorithm descriptions in the evaluation prompt increased diversity scores for identical recommendations by up to 0.89 points (Cohen’s d = 1.82, p < 10− 18), with substantial variation across models. Our results show that contextual information unrelated to recommendation quality can bias LLM-based evaluation, motivating protocols that hide algorithm metadata from LLM judges or apply mitigation strategies.