PrepRet: Automated Data Preparation Pipeline Selection for Neural Retrieval
TL;DR
PreRet, a framework that jointly optimizes preprocessing pipeline selection and neural retrieval through differentiable optimization through differentiable optimization, is proposed, enabling end-to-end gradient-based learning over a search space of 700 configurations spanning text cleaning, pre-tokenization, chunking, and augmentation.
Abstract
Neural retrieval models typically rely on fixed, hand-crafted preprocessing pipelines designed independently of the retrieval task, leading to suboptimal performance that varies across datasets and architectures. We propose PrepRet, a framework that jointly optimizes preprocessing pipeline selection and neural retrieval through differentiable optimization. We formulate preprocessing selection as a differentiable discrete choice problem using Gumbel-Softmax relaxation, enabling end-to-end gradient-based learning over a search space of 700 configurations spanning text cleaning, pre-tokenization, chunking, and augmentation. A hierarchical selection mechanism captures inter-stage dependencies between preprocessing operations. On MS MARCO, PrepRet achieves 0.533 nDCG@10, improving over Grid Search by 7.7% and Contriever by 11.7%, while requiring only 1.1 GPU-hours. Zero-shot evaluation on eight BEIR datasets confirms robust cross-domain generalization, with particularly strong gains on scientific and entity-rich domains.