Skip to content
Log in

Author

Danula Hettiachchi

View on Semantic Scholar

2 papers indexed here

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

Preprint Aug 2026

The"Curse of Knowledge"in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion

LLM-generated search queries are widely used to augment IR evaluation, yet they may contain concepts that presuppose answer-side document knowledge, violating the information-access boundary of pre-search users. Existing validation metrics, including overlap, diversity, and effectiveness, cannot distinguish rare human-tail variation from candidate answer-side intrusion. We introduce concept provenance, a framework that assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones, operationalizing a boundary that retrieval metrics alone cannot detect. Applying concept provenance to 77,004 queries across 100 UQV100 topics, 8 LLMs, and 5 prompt conditions with two extraction pipelines, we obtain a cross-pipeline token-HCIR Spearman rho of 1.0 over five condition means. Candidate answer-side concepts constitute 7.40 percent of non-generic concepts and appear in 97 of 100 topics, with topic explaining approximately 67 percent of variance. Human validation yields 68.2 percent relaxed precision, revealing two mechanisms: knowledge intrusion at 45.5 percent and deployment intrusion at 45.0 percent. Diagnostic probes show disproportionate localized retrieval effects, with deletion effect size d = -0.47 compared with d = -0.34 for random deletion, but these concepts explain less than 2 percent of aggregate evaluation variance. Concept provenance therefore serves as a boundary-compliance diagnostic rather than an evaluation-shift predictor. Under the tested conditions, no prompt condition eliminates intrusion; post-generation concept-provenance selection achieves 99 percent elimination.

Chenglong Ma, Xinye Wanyan, Danula Hettiachchi et al. · 0 citations
Book Open access Jul 2026

Reasoning with Large Language Models for Relevance Judgements

Relevance judgements are central to information retrieval (IR) evaluation, and there is much interest in using large language models (LLMs) for these judgements. This work presents a novel framework that operationalizes explicit reasoning as a control mechanism for LLM-based relevance judgements and systematically quantifies its effects across evaluation settings. We compare judgements with and without reasoning to assess the impact on accuracy and reliability across multiple datasets and models (e.g., gpt-oss, Gemini 2.5). Our results show that using reasoning often leads to lower-quality judgements and a higher rate of formatting failures, while incurring substantially higher token costs. These findings raise important questions about when and how reasoning should be incorporated into evaluation pipelines, particularly for passage-level relevance assessment, suggesting that its benefits depend on task context and complexity rather than being universally beneficial. This work also contributes a novel dataset of LLM reasoning outputs, augmenting two established test collections to support future research.

Louis Geiger, Danula Hettiachchi, Falk Scholer et al. · 0 citations