Cost-Sensitive Listwise Optimisation for Sensitivity-Aware Search

Abstract

In typical search tasks, the goal is to return a list of k documents that are ordered by their relevance to a user-issued query. However, not all information that is contained in a collection should be made available to users i.e., sensitive information should not be revealed. Sensitivity-Aware Search (SAS) is a bi-objective retrieval task that aims to make collections that potentially contain sensitive information safely searchable by producing a ranked list of documents that are (1) relevant to the query and (2) non-sensitive. Previous work on SAS leveraged pointwise relevance-only loss functions in combination with SAS-specific negative sampling techniques. We argue that modelling SAS in a pointwise manner at training time limits how effectively the task can be expressed to the model. Hence, we investigate listwise loss functions for SAS. Further, we propose a sensitivity-aware listwise distillation loss function that penalises the model for highly ranking sensitive documents during training. We perform experiments on two test collections labelled for relevance and different types of sensitivity, and observe that listwise loss functions outperform pointwise and pairwise losses, retrieving up to 58% fewer sensitive documents. Our proposed sensitivity-aware listwise distillation loss function outperforms a relevance-only listwise loss function by 12% and a pointwise loss function leveraging SAS training data by 15% in relevance-sensitivity tradeoff metrics. We release model checkpoints with PyTerrier integrations on HuggingFace.