Skip to content
Review Open access

Evaluating reasoning-tuned large language models for clinical decision-making in spine surgery.

Sep 2026 · Spine Deformity · 0 citations · 18 references
Medicine

TL;DR

OpenAI o1 was rated higher, significantly only for diagnostic accuracy and clarity, most of it reflecting presentation rather than clinical content, than DeepSeek R1, which produced plans surgeons usually rated good or excellent.

Abstract

Purpose

Most clinical evaluations of large language models assess factual recall rather than the multi-step reasoning behind operative plans. Reasoning-tuned models, post-trained to generate explicit intermediate reasoning before answering, may better approximate surgical decision-making. We compared two such models from different development ecosystems, OpenAI o1 and DeepSeek R1, on complex spine cases.

Methods

Ten synthetic vignettes spanning deformity, degenerative, urgent, and cervical pathology were analysed; an eleventh was excluded owing to a survey error. Both models received identical prompts. Outputs were anonymised and randomised. Eight fellowship-trained spine surgeons (five consultants, three fellows) scored diagnostic accuracy, reasoning and thoroughness, surgical plan appropriateness, and clarity on 5-point Likert scales, giving 78 paired evaluations. Analysis used linear mixed-effects models with crossed random intercepts for rater and vignette (Bonferroni-adjusted alpha 0.0125); reliability was assessed by intraclass correlation.

Results

o1 scored higher in all four domains, significantly for diagnostic accuracy (4.6 vs. 4.3; mean difference 0.24, 95% CI 0.11-0.38) and clarity (4.4 vs. 4.2; 0.26, 0.07-0.44); reasoning and thoroughness (0.21, 0.03-0.38), and surgical plan appropriateness (0.17, - 0.03 to 0.36) did not meet the adjusted threshold. Inter-rater reliability was poor (ICC 0.03-0.07). Clarity correlated strongly with the clinical domains (rho 0.74-0.79); adjusting for clarity attenuated the o1 advantage by 55% to over 100%, leaving no detectable clinical-domain difference.

Conclusion

Both models produced plans surgeons usually rated good or excellent. o1 was rated higher, significantly only for diagnostic accuracy and clarity, most of it reflecting presentation rather than clinical content. Surgeon oversight and format-normalised evaluation designs remain essential.

Read PDF

Similar papers

Open access Oct 2026

Large Language Models and Applied Surgical Anatomy: Evaluation of Anatomical Reasoning in Expert-Derived Plastic Surgery Scenarios.

Large language models (LLMs) are increasingly being integrated into medical education, research, and clinical decision-support applications; however, their ability to apply anatomical knowledge in clinically relevant surgical scenarios remains insufficiently studied. This study evaluated and compared the performance of...

Menekşe Kastamoni Başkan, Suleyman Cecen, Y. Kastamoni · 0 citations
Apr 2026

Large Language Models versus Expert Response: A Comparative Analysis of Patient Education Quality after Skull Base Surgery

Abstract Objective Patients undergoing endoscopic skull base surgery (ESBS) often seek postoperative guidance. Large language models (LLMs) are increasingly accessible resources, but their quality in this setting remains poorly studied. This study compared LLM-generated responses with expert responses for common postop...

M. Manavi, Noel F. Ayoub, Arifeen S. Rahman et al. · 0 citations
Review Open access Oct 2026

Personalized Decision-Making in Spine Surgery: From Predictive Algorithms to Clinical Implementation

Spine surgery has advanced in technical precision, yet recovery remains heterogeneous and is only partly explained by anatomical findings. This narrative review examines how personalized decisions can be advanced by integrating multidimensional patient data with predictive models. A structured PubMed/MEDLINE was conduc...

V. Gilard, R. Gauthé · 0 citations
Open access Oct 2026

Large language models for complex decision-making in colorectal cancer with peritoneal metastases

Background Large language models (LLMs) are increasingly evaluated for oncology decision support, but their performance in colorectal cancer with peritoneal metastases (CRC-PM) remains uncertain. We benchmarked 10 contemporary LLMs against an expert-consensus framework. Methods Forty-eight synthetic vignettes were deri...

Zhong-Wei Qi, Dong-Dong Wang, Shu-Xia Liang et al. · 0 citations
Open access Oct 2026

Large language models match and occasionally surpass the SRS AIS handbook in adolescent idiopathic scoliosis education.

BACKGROUND Patients increasingly seek medical information from artificial intelligence (AI) tools. The accuracy and clinical alignment of large language models (LLMs) relative to established, surgeon-curated patient education resources remain incompletely characterized. PURPOSE To compare LLM-generated responses with...

Prudhvi Kodali, Maveric Abella, D. Skaggs et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.