Domain-Specific Reranking: When is Adaptation Worth the Cost?
TL;DR
It is found that when pretraining data is limited, task-specific fine-tuning—using only a few hundred labeled examples in the domain—can match or surpass the gains of domain-adaptive pretraining, suggesting that domain-adaptive pretraining may not always be necessary for strong domain performance.
Abstract
Two-stage retrieval pipelines, where a fast retriever identifies candidates and then a more powerful reranker scrutinizes them, have become standard across retrieval-based applications, including open-domain question answering and retrieval-augmented generation systems. In specialized domains or low-resource languages, domain adaptation is often required to ensure retrieval quality. In this paper, we explore domain adaptation of rerankers using unlabeled and labeled, real and synthetic datasets across legal, biomedical, and auditing domains. In the domains we study, we find that when pretraining data is limited, task-specific fine-tuning—using only a few hundred labeled examples in the domain—can match or surpass the gains of domain-adaptive pretraining, suggesting that domain-adaptive pretraining may not always be necessary for strong domain performance. We further evaluate this on two industrial Japanese datasets, where we observe practical improvements with purely synthetic training data.