SpineXR-VQA: A Clinically-Validated Visual Question Answering Dataset for Spine X-Rays
TL;DR
This resource paper introduces SpineXR-VQA, an open-source, clinically grounded, and verified benchmark comprising 2,187 X-rays and 8,272 expert-verified, open-ended Question-Answer pairs that benchmark 15 state-of-the-art Multimodal Large Language Models (MLLMs), including proprietary systems such as Claude Sonnet, GPT-o4 mini, and Gemini Flash.
Abstract
Recent progress in Medical Visual Question Answering (VQA) has significantly aided clinical decision-making across various domains such as pneumonia, oncology, and neurology. However, spinal and musculoskeletal ailments remain critically underexplored. The development of reliable VQA models for spinal imaging is currently hindered by a lack of datasets and evaluation protocols that do not reflect the descriptive, diagnostic interpretations used in clinical practice. In this resource paper, we introduce SpineXR-VQA, an open-source, clinically grounded, and verified benchmark comprising 2,187 X-rays and 8,272 expert-verified, open-ended Question-Answer pairs. Unlike standard classification-based datasets, SpineXR-VQA features six expert-validated categories: abnormality, severity, location, diagnosis, treatment, and reasoning. Ten orthopedic specialists from India and Thailand validated all pairs, ensuring geographic diversity and high inter-rater agreement (Cohen's Kappa: 0.96 for questions, 0.93 for answers). We benchmark 15 state-of-the-art Multimodal Large Language Models (MLLMs), including proprietary systems such as Claude Sonnet, GPT-o4 mini, and Gemini Flash, alongside medical-specific and general open-weight models. While these models achieve moderate semantic alignment (median similarity: 0.72), a detailed analysis reveals that they consistently fail to capture essential diagnostic nuances, such as anatomical fidelity and clinical completeness. These results underscore the necessity for specialized models in spinal VQA, a gap SpineXR-VQA fills.