OpenAI o1 was rated higher, significantly only for diagnostic accuracy and clarity, most of it reflecting presentation rather than clinical content, than DeepSeek R1, which produced plans surgeons usually rated good or excellent.
Abstract
Purpose
Most clinical evaluations of large language models assess factual recall rather than the multi-step reasoning behind operative plans. Reasoning-tuned models, post-trained to generate explicit intermediate reasoning before answering, may better approximate surgical decision-making. We compared two such models from different development ecosystems, OpenAI o1 and DeepSeek R1, on complex spine cases.
Methods
Ten synthetic vignettes spanning deformity, degenerative, urgent, and cervical pathology were analysed; an eleventh was excluded owing to a survey error. Both models received identical prompts. Outputs were anonymised and randomised. Eight fellowship-trained spine surgeons (five consultants, three fellows) scored diagnostic accuracy, reasoning and thoroughness, surgical plan appropriateness, and clarity on 5-point Likert scales, giving 78 paired evaluations. Analysis used linear mixed-effects models with crossed random intercepts for rater and vignette (Bonferroni-adjusted alpha 0.0125); reliability was assessed by intraclass correlation.
Results
o1 scored higher in all four domains, significantly for diagnostic accuracy (4.6 vs. 4.3; mean difference 0.24, 95% CI 0.11-0.38) and clarity (4.4 vs. 4.2; 0.26, 0.07-0.44); reasoning and thoroughness (0.21, 0.03-0.38), and surgical plan appropriateness (0.17, - 0.03 to 0.36) did not meet the adjusted threshold. Inter-rater reliability was poor (ICC 0.03-0.07). Clarity correlated strongly with the clinical domains (rho 0.74-0.79); adjusting for clarity attenuated the o1 advantage by 55% to over 100%, leaving no detectable clinical-domain difference.
Conclusion
Both models produced plans surgeons usually rated good or excellent. o1 was rated higher, significantly only for diagnostic accuracy and clarity, most of it reflecting presentation rather than clinical content. Surgeon oversight and format-normalised evaluation designs remain essential.
Large language models (LLMs) are increasingly being integrated into medical education, research, and clinical decision-support applications; however, their ability to apply anatomical knowledge in clinically relevant surgical scenarios remains insufficiently studied. This study evaluated and compared the performance of...
Menekşe Kastamoni Başkan, Suleyman Cecen, Y. Kastamoni· Clinical anatomy (New York,...· 0 citations
Abstract Objective Patients undergoing endoscopic skull base surgery (ESBS) often seek postoperative guidance. Large language models (LLMs) are increasingly accessible resources, but their quality in this setting remains poorly studied. This study compared LLM-generated responses with expert responses for common postop...
M. Manavi, Noel F. Ayoub, Arifeen S. Rahman et al.· Skull Base· 0 citations
Spine surgery has advanced in technical precision, yet recovery remains heterogeneous and is only partly explained by anatomical findings. This narrative review examines how personalized decisions can be advanced by integrating multidimensional patient data with predictive models. A structured PubMed/MEDLINE was conduc...
V. Gilard, R. Gauthé· Journal of Personalized Medi...· 0 citations
Background Large language models (LLMs) are increasingly evaluated for oncology decision support, but their performance in colorectal cancer with peritoneal metastases (CRC-PM) remains uncertain. We benchmarked 10 contemporary LLMs against an expert-consensus framework. Methods Forty-eight synthetic vignettes were deri...
Zhong-Wei Qi, Dong-Dong Wang, Shu-Xia Liang et al.· Frontiers in Medicine· 0 citations
BACKGROUND
Patients increasingly seek medical information from artificial intelligence (AI) tools. The accuracy and clinical alignment of large language models (LLMs) relative to established, surgeon-curated patient education resources remain incompletely characterized.
PURPOSE
To compare LLM-generated responses with...
Prudhvi Kodali, Maveric Abella, D. Skaggs et al.· Spine Deformity· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.