Skip to content
Review Open access

Automating Abstract Screening in Research Synthesis Using Large Language Models: A Tutorial and Proof-of-Concept Study

Oct 2026 · Advances in Methods and Practices in Psychological Science · 0 citations · 55 references

Abstract

Screening abstracts is a crucial yet labor-intensive step in research-synthesis projects, such as systematic reviews and meta-analyses. Large language models (LLMs) promise an opportunity to streamline and automate this process. However, there is still little guidance and practical insight into how such automated workflows can be implemented in research practice. In this article, we propose, describe, and illustrate an abstract-screening procedure using LLMs including a quantification of uncertainty of model outputs. We provide a step-by-step tutorial in which we address how to select an LLM, access LLMs from within R, develop and refine suitable prompts, define structured output formats, and describe whether and how model hyperparameters should be set. We then illustrate this procedure in a proof-of-concept study using a set of human-rated abstracts from a recent meta-analysis for comparison. We demonstrate that LLM-assisted screening can substantially reduce the time and cost of review preparation while maintaining accuracy comparable with human raters. To support psychological researchers in adopting LLM-based workflows for research synthesis, we also provide R scripts and implementation guidance on Gitlab. At the same time, we emphasize that this work represents an initial step and that continued refinement and validation are essential given that LLM technologies and their applications continue to evolve rapidly.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.