Leveraging Large Language Model Token Probabilities for Literature Review Abstract Filtering

Abstract

Grasping the latest knowledge in any field has become increasingly challenging given the rapid pace of modern publishing. Recent advances in artificial intelligence (AI), particularly embedding models and large language models (LLMs), provide promising tools to aid researchers in the filtration and identification of the most relevant literature. However, there is little practical guidance on how to incorporate LLMs into literature reviews, and there are not yet direct comparisons of embedding models versus generative approaches. In this work, we compare these methods using small local LLMs for literature screening on both a simple dataset containing distinct categories and a more complex example with abstracts obtained from the Web of Science. We find that embedding models perform strongly for simple, well-separated queries due to their robustness and computational efficiency. For more complex filtering tasks, generative approaches based on token probabilities can achieve competitive or superior performance, but only when properly calibrated and when token-level outputs are interpreted as continuous scores rather than binary decisions. However, we show these methods are highly sensitive to prompt formulation, token conventions, and threshold selection.