Enhancing Web Search Effectiveness and Robustness Using BERT- and Co-Occurrence-Based Pruned Conflation Sets

Abstract

Stemming is a fundamental technique for enhancing recall in Web search systems. It allows documents containing morphological variants of query terms to be included in the result set. However, rule-based stemmers produce large conflation sets in which many variants are unrelated to the query intent, and appending them at query time degrades both effectiveness and efficiency. This paper introduces a method for pruning the conflation sets of query terms. A contextual similarity signal obtained from BERT is rescaled at the distribution level, then combined with normalized pointwise mutual information and thresholded. Rescaling is necessary. The contextual signal cannot be thresholded directly because it is strongly anisotropic, and the similarity distributions of unrelated word pairs and of related morphological variants share roughly half their mass. Comprehensive evaluations demonstrate that pruning improves retrieval effectiveness over the unpruned conflation sets, and that a single fixed parameter configuration selected on one collection transfers to the other with a loss of about 1.5% relative to the best attainable configuration. A risk-sensitive per-query analysis shows that the pruned systems are more robust than their base stemmers. The pruning also reduces the number of terms submitted per query by more than 92% relative to the full conflation set.