Skip to content
Open access

Large language models for complex decision-making in colorectal cancer with peritoneal metastases

Oct 2026 · Frontiers in Medicine · Vol 13 · 0 citations · 34 references
Medicine

Abstract

Background Large language models (LLMs) are increasingly evaluated for oncology decision support, but their performance in colorectal cancer with peritoneal metastases (CRC-PM) remains uncertain. We benchmarked 10 contemporary LLMs against an expert-consensus framework. Methods Forty-eight synthetic vignettes were derived from a 2026 CRC-PM consensus guideline, including 36 primary scenarios and 12 counterfactual scenarios forming six pairs. Each model completed every vignette three times using a frozen zero-shot structured prompt. The primary endpoint was prompt-aligned structured Full concordance; contradiction-only and full-semantic adjudication were sensitivity analyses. Two blinded attending colorectal surgeons, each with 5 years of clinical experience, independently validated the reference classifications. Results Among 1,080 primary responses, structured scoring classified 609 (56.4%) as Full, 117 (10.8%) as Partial, and 354 (32.8%) as Discordant. Model-specific structured Full concordance ranged from 49.1% to 63.0%; neither the Friedman test (P = .104) nor clustered GEE (P = .186) supported a robust model effect. Contradiction-only scoring was similar (56.2% Full), whereas full-semantic adjudication reduced Full concordance to 45.5%. Excluding prompt-development scenarios lowered absolute concordance but did not change the inferential conclusion. Expert agreement was 93.8%–97.9% across decision domains (Cohen kappa 0.898–0.948). Counterfactual and metachronous risk-management analyses showed wide uncertainty, and 21 clinically consequential discordance flags arose from five scenarios. Conclusions LLM performance in CRC-PM was sensitive to scoring definition and did not establish a stable between-model hierarchy. Prospective clinician-plus-LLM studies are required before clinical benefit can be inferred.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.