The Plug-and-Play Illusion: Cross-Encoder Brittleness to Retrieval Noise under Fixed Recall
TL;DR
The conclusion is not that LLM rerankers are robust, but that negative distributions are part of the retriever-reranker interface and should be checked before deployment.
Abstract
Cross-encoder rerankers are often treated as safe, drop-in quality boosts once a first-stage retriever has recalled a candidate pool. The hidden assumption is simple: if the gold document is present, the reranker should still pick it out from whatever negatives the retriever supplies. We test that assumption with a recall-controlled stress test that keeps recall at 100% and varies only the distractor distribution. Under this protocol, a small cross-encoder's NDCG@10 falls by up to 50% when negatives shift from random documents to retriever-mined hard negatives. The collapse repeats across four rerankers spanning 4M-278M parameters and two training families, so it is not one weak model. We then ask whether zero-shot LLM rerankers fail in the same way. A listwise language-model reranker, with gpt-4o-mini as the strongest case, is less sensitive to BM25 hard negatives across three BEIR corpora, but the controls complicate the story. On nfcorpus, the advantage relies on the BM25-seeded listwise order. On fiqa and scifact, it survives pointwise scoring and shuffled-order controls, indicating intrinsic model robustness. Dense hard negatives still hurt all rerankers. The conclusion is therefore not that ''LLMs are robust,'' but that negative distributions are part of the retriever-reranker interface and should be checked before deployment. We end with a no-new-label diagnostic.
