Skip to content

Multi‐Modal Transformer Framework for Automated Knee Osteoarthritis Severity Grading From Radiographs

Sep 2026 · International journal of imaging systems and technology (Print) · Vol 36 · 0 citations · 31 references

TL;DR

A multi‐modal, artificial intelligence (AI)‐driven framework for automated grading of knee osteoarthritis severity from radiographs, integrating advanced detection, deep learning, radiomics, and clinical data is developed and validated.

Abstract

To develop and validate a multi‐modal, artificial intelligence (AI)‐driven framework for automated grading of knee osteoarthritis severity from radiographs, integrating advanced detection, deep learning, radiomics, and clinical data. A total of 2290 patients from five medical centers were included, split into an internal cohort ( n  = 1549) and an external test set ( n  = 741). Knee radiographs were preprocessed with histogram normalization, contrast‐limited adaptive histogram equalization (CLAHE), and noise filtering. Four detection models, RT‐DETR, SAM, DINOv2 + ViTDet, and Grounded‐SAM/MedSAM, were evaluated for anatomical localization, with Grounded‐SAM/MedSAM selected for optimal intersection over union (IoU) and mean average precision (mAP). Radiomic features ( n  = 215) were extracted from detected regions of interest (ROIs) using the Standardized Environment for Radiomics Analysis (SERA) platform, while deep features were derived from four advanced extractors, with MedSAM + feature aggregator selected for highest classification accuracy. Features underwent harmonization, variance filtering, and selection via least absolute shrinkage and selection operator (LASSO), principal component analysis (PCA), or mutual information. Clinical, radiomic, and deep features were fused and classified using CoAtNet, ConvNeXt V2, EfficientNetV2, RepLKNet, and Swin V2 Transformers. Model selection was performed within a nested cross‐validation scheme on the internal cohort, and the external cohort was accessed only once after the final configuration had been fixed. Model performance was assessed by accuracy, F1‐score, and area under the receiver operating characteristic curve (AUC), with interpretability provided by SHapley Additive exPlanations (SHAP) analysis and a nomogram. Grounded‐SAM/MedSAM achieved the highest detection performance (external IoU = 90.83% [95% confidence interval, CI, 89.41–92.15], mAP = 93.67% [95% CI 92.38–94.82]). MedSAM deep features outperformed others (external F1 = 82.37% [95% CI 80.56–84.18], AUC = 0.902 [95% CI 0.888–0.916]). Fused features with Swin V2 + LASSO yielded the best results (external accuracy = 94.6% [95% CI 93.5–95.7], F1 = 94.4% [95% CI 93.3–95.5], AUC = 97.2% [95% CI 96.4–98.0]), significantly surpassing single‐modality models. SHAP analysis confirmed balanced contributions from deep, radiomic, and clinical features, while the nomogram achieved a C‐index of 0.970 [95% CI 0.958–0.980] with excellent calibration. This framework demonstrates high accuracy, robustness, and interpretability in knee osteoarthritis severity grading from radiographs, offering strong potential for real‐world clinical integration.

View source

Similar papers

Review Sep 2026

A comparative study of machine learning based automatic diagnosis approaches to diagnose knee osteoarthritis from radiographs.

BACKGROUND Knee Osteoarthritis (KOA) is a common degenerative joint condition that affects millions of middle-aged and elderly people worldwide, mainly due to the gradual loss of articular cartilage. While diagnosis often depends on X-ray imaging, grading KOA severity with Kellgren-Lawrence (KL) scales is subjective an...

Satish Chandra, Priyadarshini, Vishwambhar Pathak · 0 citations
Open access Oct 2026

ROI-Aware Ordinal Multi-Task Framework for Knee Osteoarthritis Severity Assessment from X-Ray Images

An ROI-aware ordinal and uncertainty-aware multi-task framework for automated KOA severity assessment that integrates ROI-guided feature learning, ordinal KL-grade classification, and severity regression to better model progressive disease characteristics is proposed.

T. Phạm, Thanh Luan Nguyen · 0 citations
Open access Sep 2026

Lightweight Vision Transformer Ensemble With External Validation For Binary Knee Osteoarthritis Detection

Knee osteoarthritis (KOA) is another degenerative joint disorder that has a profound effect on mobility and quality of life especially with aging populations. Timely detection may rely on early and accurate detection, yet the traditional method of grading radiological images by manual means is subjective, non-objective...

Bhimisetty Chandana Sri, V. S. Narayana Tinnaluri · 0 citations
Open access Sep 2026

Deep learning-based multiclass classification for the diagnosis of hand arthritis on digital radiography: a multicenter study

The transformer-based model showed strong internal performance and retained useful discriminative ability in an independent external cohort; however, the reduction in external accuracy and macro-F1 indicated limited cross-center transferability.

Chao Deng, Dan Yan, Cong Tang et al. · 0 citations
Review Open access Oct 2026

Development and Validation of an Interpretable Deep Learning Model for Automated Kellgren–Lawrence Grading of Knee Osteoarthritis Using X‐Ray Imaging

Accurate and reproducible Kellgren–Lawrence grading of radiographic knee osteoarthritis remains challenging, particularly for intermediate grades and anatomically heterogeneous compartments. We present X‐VIG, an interpretable deep learning framework integrating paired anteroposterior and lateral knee radiographs via vi...

Zhen-Bang Dai, Cheng-Cheng Feng, Meng-Jie Ni et al. · 0 citations
Open access Sep 2026

Fast Severity Ordered Boundary Residual Learning for Multi-Class Knee Osteoarthritis Grading from X-Ray Images

Objectives: To develop a model for the classification of knee OsteoArthritis (OA) on radiographs, which is computationally efficient and severity ordered. Method: This study focuses on class imbalance, adjacent-grade confusion and high computational cost of multi-model hybrid systems. The proposed SOBR-KneeNet (Severit...

S. Bruntha, M. Chidambaram · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.