Skip to content
Open access

Large language models match and occasionally surpass the SRS AIS handbook in adolescent idiopathic scoliosis education.

Oct 2026 · Spine Deformity · 0 citations · 15 references
Medicine

Abstract

Background

Patients increasingly seek medical information from artificial intelligence (AI) tools. The accuracy and clinical alignment of large language models (LLMs) relative to established, surgeon-curated patient education resources remain incompletely characterized.

Purpose

To compare LLM-generated responses with the Scoliosis Research Society (SRS) Adolescent Idiopathic Scoliosis (AIS) Handbook using a surgeon-adjudicated alignment scoring system. STUDY

Design

Questionnaire-based comparative study.

Methods

Four LLMs (Gemini 3, Claude Sonnet 4.5, GPT-5.2, and Grok 4.1) were prompted using all 19 SRS AIS frequently asked questions (FAQs) items. Three fellowship-trained spine surgeons, blinded to model identity, independently evaluated 76 LLM responses using a novel 6-point alignment scale assessing accuracy, completeness, and concordance with SRS guidance. Responses contradicting the handbook but judged more accurate by surgeon consensus received the maximum score of 6. Inter-rater reliability was assessed at the item and model level using intraclass correlation coefficients (ICC). Model performance was compared using repeated-measures analysis of variance (ANOVA) with Holm-adjusted pairwise comparisons.

Results

Mean scores (95% CI) were 5.23 (5.04-5.41) for Grok 4.1, 4.91 (4.66-5.16) for Gemini 3, 4.75 (4.44-5.07) for GPT-5.2, and 4.51 (4.25-4.77) for Claude Sonnet 4.5 (F (3, 54) = 7.63, p<0.001). Inter-rater reliability was moderate at the item level (ICC [2,1] = 0.45; ICC [2,3] = 0.71) and excellent at the model level (ICC [2,1] = 0.82; ICC [2,3] = 0.93). Overall, 82.5% of ratings (188/228) achieved scores ≥5, indicating strong concordance with or extension beyond guideline recommendations. Notably, 12.3% of ratings (28/228) received the maximum score of 6, reflecting surgeon consensus that the LLM output contradicted but exceeded the SRS handbook in accuracy. No ratings were graded as completely incorrect.

Conclusions

Modern LLMs generate AIS patient education content that matches and occasionally surpasses the SRS AIS Handbook. Discrepancies between AI-generated responses and guideline-based materials may reflect evolving clinical evidence and highlight opportunities for updating patient-facing resources. Because LLMs increasingly reference authoritative educational materials, updating the SRS Handbook would directly enhance both traditional and AI-mediated patient education.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.