Skip to content
Review Open access

Zero-Shot Per-Criteria Prompting for Systematic Review Acceleration: A Case Study of Title and Abstract Filtering Process

Oct 2026 · Journal of Information Systems Engineering and Business Intelligence · Vol 12, pp. 760-774 · 0 citations · 28 references

TL;DR

This study investigates the potential of large language models (LLMs) to automate the document selection phase in the SLR process by proposing a per-criteria prompting approach that naturally mimics the human annotation process, thereby improving screening efficiency and accuracy.

Abstract

Background: Systematic Literature Review (SLR) is a widely adopted method for obtaining a comprehensive understanding of a specific research area. It involves a structured, transparent, and research question-driven process consisting of three main stages: planning, conducting, and reporting. Although SLR provides objectivity and accuracy, the document selection phase remains labor-intensive and time-consuming due to the need to manually screen large volumes of literature. Objective: This study investigates the potential of large language models (LLMs) to automate the document selection phase in the SLR process by proposing a per-criteria prompting approach that naturally mimics the human annotation process, thereby improving screening efficiency and accuracy. Methods: We propose a per-criteria prompting approach to classify paper metadata using LLM in the zero-shot paradigm. This is done by classifying whether the given paper metadata (title and abstract) is relevant to each inclusion exclusion criteria and then aggregating them. As the baseline, we compare our proposed method to the existing global prompting approach, which directly classifies whether the given paper metadata is relevant to all inclusion exclusion criteria in single steps. Results: The proposed per-criteria prompting approach generally outperforms the global prompting approach. Especially when applied using the Llama model, our proposed approach outperforms the existing global prompting approach in all 10 dataset benchmarks we use. Conclusion: The finding suggests that prompting each criterion significantly boosts the LLM performance because it makes the LLM more focused in classifying the paper metadata into the given inclusion and exclusion criteria. Keywords: Systematic Literature Review, Text Classification, Large Language Model, Per-Criteria Prompting, Zero-Shot Prompting

Read PDF

Similar papers

Review Open access Nov 2026

A retrieval-augmented LLM framework for PRISMA-aligned systematic reviews: a case study on action quality assessment

A framework that integrates Large Language Models (LLMs) and retrieval-augmented generation (RAG) into the PRISMA 2020 workflow to assist and standardize key review tasks and is structured to facilitate adaptation to other research areas by declaring new query and prompt templates.

Romeo Silvestri, Miguel Pincheira, Massimo Vecchio et al. · 0 citations
#explainable ai Book Open access Oct 2026

Evaluating and explaining LLM classification for automatic screening in Systematic Reviews

This research conducts a set of experiments using open-source Large Language Models with small parameters hosted on local servers to screen titles and abstracts against pre-defined eligibility criteria respective to each topic, and suggests that AI may indeed aid with initial relevance screening, but more research is n...

Min-Ji Kang, Bipasha Banerjee, C. Comer · 0 citations
Review Open access Oct 2026

Evaluating prompting strategies and large language models in systematic literature review screening: Relevance and task-stage classification.

This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, fe...

Bing-Lan Han, A. Mathrani, Vajisha U. Wanniarachchi et al. · 1 citation
Review Open access Oct 2026

Automating Abstract Screening in Research Synthesis Using Large Language Models: A Tutorial and Proof-of-Concept Study

Screening abstracts is a crucial yet labor-intensive step in research-synthesis projects, such as systematic reviews and meta-analyses. Large language models (LLMs) promise an opportunity to streamline and automate this process. However, there is still little guidance and practical insight into how such automated workf...

Mirka Henninger, J. Radek, Jean-Paul Snijder-Steinhilber et al. · 0 citations
Open access 2026

Comparative Evaluation of Prompt Engineering Configurations and Open-Source Large Language Models for Phishing Email Detection

Overall, the results show that prompt configuration and model choice jointly influence phishing detection performance, false-alarm rates, output reliability, and local processing efficiency.

Naavin A/L Rajeswaren, Agus Hartoyo, Farhad Nadi · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.