Application of LLM combined with knowledge base in improving C-TIRADS grading of thyroid nodules

Abstract

Objective This study used the unified text-based grading results of senior sonographers as the primary reference (this text benchmark is not pathological diagnostic gold standard) to evaluate the concordance of text-derived C-TIRADS grades assigned by six large language models (LLMs), with and without FastGPT Retrieval-Augmented Generation (RAG) knowledge base assistance. Our primary endpoint was the concordance between grading and physician consensus—a measure fundamentally distinct from pathological diagnostic accuracy. Only a small subgroup with pathological confirmation was used for a supplementary exploratory analysis of the correlation between model risk stratification and pathological benign or malignant findings. Methods Six LLMs were configured using the C-TIRADS criteria. A FastGPT knowledge base was built from 11,489 thyroid ultrasound records collected at Yantai Yuhuangding Hospital between 2022 and 2023. Cases were assigned to manual, LLM, and knowledge-base-augmented LLM groups. C-TIRADS grading concordance with expert ultrasound consensus, and grading speed were assessed. A subgroup of 62 pathologically confirmed nodules was used for supplementary exploratory analysis. AUC, sensitivity, specificity, PPV, NPV and Mantel-Haenszel linear trend χ2 were computed to examine the alignment of model C-TIRADS risk tiers with pathological benign and malignant findings. Results Among the standalone LLMs, DeepSeek-V3 achieved the highest numerical grading concordance (81.14%). Following the integration of the RAG knowledge base, overall grading consistency improved significantly ( P < 0.001), with Doubao-pro-128k demonstrating the highest overall concordance at 89.04%. Concordance increased significantly for C-TIRADS Categories 1 and 3 ( P = 0.027 and P < 0.001, respectively), increased numerically for Category 2 ( P = 0.596), and decreased numerically for borderline Category 4A ( P = 0.106). The average concordance across all categories rose significantly from 70.39% to 79.75% ( P < 0.001). The knowledge-augmented Doubao-pro-128k model exhibited excellent inter-rater agreement ( linear-weighted Kappa>0.8, Bonferroni-corrected P<0.006 ). The average single-case grading duration of all 12 AI risk tiering pipelines (6 LLMs with/without knowledge base) was 1.30 s, which was 15 times faster than manual grading (19.25 s). Stratified trend and ROC analyses were performed on 62 thyroid nodules with pathological outcomes. Within this small-sample exploratory pathological subgroup, the Doubao-pro-128k integrated with knowledge base established a significantly ascending malignant risk gradient, achieving an AUC of 0.783 (95% CI : 0.635, 0.932) for distinguishing benign from malignant lesions. In comparison, knowledge-enhanced Doubao-pro-128k exhibited a trend toward better matching trend with pathological malignancy distribution and numerically the highest positive Youden index of 0.277 (95% CI : 0.071, 0.482). Conclusion Doubao-pro-128k integrated with knowledge base accelerated text grading and improved overall sonographer concordance; category-specific concordance increased significantly for C-TIRADS 1 and 3, increased numerically for Category 2, and declined numerically without statistical significance for borderline C-TIRADS 4A, indicating that these ambiguous nodules require further optimization. Exploratory pathological analysis revealed variable model-pathology alignment at the C-TIRADS ≥ 4A cutoff: DeepSeek-V3 integrated with knowledge base produced a negative Youden index, while Doubao-pro-128k integrated with knowledge base showed a tentative trend toward better reflected pathological malignancy gradients.