Skip to content
Review Open access

Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study

Oct 2026 · medRxiv · 0 citations
Medicine

TL;DR

Requiring unanimous agreement across multiple LLMs enabled high-precision automatic adjudication while directing ambiguous cases to manual review to reduce the burden of manual adjudication and support scalable construction of high-purity research cohorts from unstructured clinical text.

Abstract

Background Large-scale retrospective studies often require researchers to adjudicate outcomes or cohort eligibility from unstructured clinical records. Manual review can provide reliable labels but is difficult to scale. We developed and evaluated a consensus workflow using multiple large language models (LLMs) to automatically assign high-confidence outcome labels while deferring ambiguous cases for manual review. Methods We included 6,718 brain MRI reports from 5,856 patients. The expert-adjudicated reference set included 680 reports from 675 patients: 372 normal (age-appropriate without significant intracranial pathology), 90 white matter hyperintensity-only (WMH-only; isolated nonspecific WMHs potentially ambiguous for binary classification), and 218 abnormal (other intracranial pathology) reports. We evaluated proprietary-inclusive (Qwen3/Kimi-Linear/GPT-5) and open-source (Qwen3/Kimi-Linear/DeepSeek-R1) ensembles against individual LLMs. Ensembles assigned a normal or abnormal label only when all models agreed; all other reports were classified as uncertain and deferred for manual review. Individual LLMs assigned normal or abnormal predictions and classified WMH-only predictions as uncertain. In the full cohort, we compared the prevalence of active central nervous system (CNS)-related International Classification of Diseases, Tenth Revision (ICD-10) diagnoses across the normal, uncertain, and abnormal groups generated by each ensemble. Results In the expert-adjudicated reference set, both ensembles achieved 100% precision for automatically assigned normal and abnormal reports. GPT-5 was the strongest individual LLM but had lower precision than either ensemble, with normal and abnormal precision of 98.9% and 94.1%, respectively, and 17 expert-label mismatches versus none for the consensus ensembles. In the full cohort, 1.6% of reports classified as normal were associated with CNS-related ICD-10 diagnoses, compared with 29.7% of uncertain and 51.3% of abnormal reports for the proprietary-inclusive ensemble; corresponding rates were 1.6%, 31.1%, and 51.6% for the open-source ensemble. Conclusions Requiring unanimous agreement across multiple LLMs enabled high-precision automatic adjudication while directing ambiguous cases to manual review. This strategy may reduce the burden of manual adjudication and support scalable construction of high-purity research cohorts from unstructured clinical text.

Read PDF

Similar papers

Open access Aug 2026

Incremental Diagnostic Value of Clinical Information for Large Language Models Across Multiple Organs: Retrospective Study

The addition of clinical information was associated with a numeric trend toward higher diagnostic accuracy overall, but this trend was heterogeneous across models and disease types, and no statistically significant improvement was demonstrated after adjustment for multiple comparisons.

Jin-Qi Zhang, Xiao-Yi Wang, Yanfeng Zhao et al. · 0 citations
#small language model Open access Oct 2026

Application of large language models for assigning clear cell likelihood score v2.0 from free-text MRI reports: a feasibility study.

RATIONALE AND OBJECTIVES To evaluate the performance of four large language models (LLMs) for automated feature extraction and clear cell likelihood score version 2.0 (ccLS v2.0) assignment from free-text reports using a multi-step prompting strategy. MATERIALS AND METHODS This retrospective multicenter study include...

Yan-Ting Duan, Yi-Lei Zhao, Mao-Wei He et al. · 0 citations
Open access Oct 2026

Comparative Evaluation of Open-Weight Large Language Models for Pathological Brain MRI Reports Classification: From Prompts to a Fine-Tuned Model.

RATIONALE AND OBJECTIVES Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) i...

Claudio Demaria, Claudio Crema, Mario Verdicchio et al. · 0 citations
Review Open access Aug 2026

Error Detection and Correction in Chinese Radiology Reports Using Large Language Models: Real-World Clinical Validation Study

Enhanced LLMs, particularly DeepSeek-R1, demonstrated robust performance in error detection and correction within real-world Chinese radiology reports, supporting their clinical use for automated quality assurance and integration into workflows to improve reporting accuracy and efficiency.

Jia-Feng Zhou, Yuxin Wei, Qian Cai et al. · 0 citations
Review Open access Sep 2026

Large language models for patient-facing pathology report interpretation: A scoping review.

LLM-based patient-facing pathology report interpretation shows potential to bridge specialist pathology language and patient communication, and evaluation should extend beyond readability to include fidelity to the original pathology report, patient understanding, safety, and usability.

Chen Wang, Jie Hao, Si-Jia Zhang et al. · 0 citations
#small language model Review Open access Sep 2026

A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology

With targeted methodological refinements - including standardization of QUIPS implementation and validation against expert ratings - automated ROB assessments may meaningfully reduce time and cost of systematic reviews of prognosis studies in neurology and beyond.

S. Kampman, K. Braun, W. Otte · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.