The Missing Minimal Pair: Stereotype Evaluation in LLMs
A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To add...
N. Stepanova, I. Titov, Emily Allaway et al.
· 0 citations