The Relevance of (Relevant) Entity Presence
TL;DR
It is concluded that knowing the relevance of entities for an information need can be very valuable, but that the methods described are insufficient to determine this relevance information from the documents and queries alone.
Abstract
Query-Specific Document and Entity Representations (QDER) has shown state-of-the-art effectiveness on multiple information retrieval benchmarks. This work attempts to reproduce this promising approach. In an initial attempt, we were unable to find results that resemble the convincing effects measured in the original work. A deep dive into the published artifacts, plus a comparative study of similar work by the same authors show inconsistencies between the descriptions in the paper and their implementation in the published artifacts, and a leakage of relevance assessments due to oversampling entities that are known to appear in relevant documents only. Due to dependencies in the ranking pipeline upon relevance assessments, the final ranking models gain knowledge about the relevance of entities. We conclude that knowing the relevance of entities for an information need can be very valuable, but that the methods described are insufficient to determine this relevance information from the documents and queries alone. The resulting pipeline cannot be deployed effectively in practice, limiting the impact of the research results.