Performance of leading large language models in adhering to clinical guidelines for anaplastic thyroid cancer: a comparative study
TL;DR
Leading LLMs show variable capacity to align with ATC clinical guidelines, while top-performing models hold promise as supportive tools, their inconsistencies across domains and complexity levels preclude autonomous clinical use.
Abstract
Anaplastic thyroid cancer (ATC) is a rare, aggressive malignancy with poor prognosis. Adherence to guidelines from the National Comprehensive Cancer Network (NCCN), American Thyroid Association (ATA), and European Society for Medical Oncology (ESMO) is critical for optimal patient outcomes. As large language models (LLMs) increasingly enter clinical workflows, rigorous evaluation of their alignment with established guidelines is essential. We evaluated five leading LLMs for their ability to generate guideline-concordant responses to clinical questions about ATC. We conducted a comparative study in 2025 following TRIPOD-LLM (Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis, Large Language Models) guidelines. Seventy clinical questions of varying complexity were developed from ATA, NCCN, and ESMO guidelines. Three surgical oncology experts validated each question and subsequently evaluated responses from five LLMs: ChatGPT 4.1, ChatGPT 5, Gemini 2.5 Pro, Claude Sonnet 4, and DeepSeek R1. Each response was scored for relevancy, clarity, accuracy, and adequacy on a 5-point Likert scale. Inter-rater reliability was assessed using both intraclass correlation coefficients (ICC) and Gwet’s AC2 with ordinal weights. Model comparisons used the Kruskal-Wallis test with Dunn’s post-hoc analysis and Bonferroni correction. A pre-specified sensitivity analysis excluding the unblinded model (ChatGPT 5) was performed to confirm robustness. Significant performance differences emerged across all four metrics: accuracy (p = 0.007), adequacy (p = 0.003), clarity (p = 0.014), and relevance (p < 0.001). Gemini 2.5 Pro achieved the highest median accuracy (4.5), followed by DeepSeek R1 (4.4), while ChatGPT 4.1 scored lowest (4.0). ICC values ranged from 0.34 to 0.44 (poor to moderate), but Gwet’s AC2 yielded substantially higher estimates of 0.61 to 0.73 (moderate to substantial agreement), reflecting the impact of restricted score range on conventional reliability metrics. The sensitivity analysis excluding ChatGPT 5 confirmed the performance hierarchy among blinded models, with significance preserved or strengthened across all four metrics. Leading LLMs show variable capacity to align with ATC clinical guidelines. While top-performing models hold promise as supportive tools, their inconsistencies across domains and complexity levels preclude autonomous clinical use. These models should serve strictly as decision aids under expert supervision.