Jun 2026· Healthcare· Vol 14, pp. 1877· 2 citations· 64 references
Medicine
TL;DR
A systematic review of the diagnostic performance of multimodal VLMs integrating both patient textual and image data across diverse real-world hospital settings suggests that multimodal VLMs consistently outperform unimodal models with access to only image or text.
Abstract
Highlights What are the main findings? Multimodal diagnostic vision-language models (VLMs) consistently outperformed unimodal models using either text or image data alone. Physicians with VLM assistance achived higher diagnostic accuracy compared to physicians without VLM support. However, there is conflicting evidence for VLMs as standalone diagnostic agents. What are the implications of the main findings? The consistent superiority of multimodal VLMs implies that future diagnostic systems require architectures that can effectively process imaging and clinical text to capture the necessary context for accurate diagnosis. Given the conflicting evidence for standalone AI versus the copilot model, implementation should focus on AI-assisted workflows that augment human expertise rather than replacing it, positioning VLMs as robust clinical decision support tools to enhance workflow, mitigate physician fatigue, and reduce cognitive bias. Abstract Background/Objectives: Novel vision-language models (VLMs) can integrate patient textual data with image data to support medical diagnosis. Recent studies reported conflicting results regarding the performance of multimodal VLMs compared to other models and physician performance. This systematic review aims to assess the diagnostic performance of multimodal VLMs integrating both patient textual and image data across diverse real-world hospital settings. Methods: We performed comprehensive searches of eight resources, including Embase, MEDLINE, and SCOPUS, on 17 December 2025. Eligible studies reporting diagnostic performance of VLMs integrating both image and patient history textual data from real-world adult patients compared to that of other models and physicians were included. The review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The Prediction model study Risk Of Bias Assessment Tool + AI (PROBAST + AI) was used to assess the quality and risk of bias. The study protocol was registered in the PROSPERO database (CRD420251244054). This review received no external funding. Results: We screened 11,026 records, of which 18 studies met the inclusion criteria. Six studies comparing multimodal and unimodal models demonstrated the consistent superiority of the multimodal models. Four studies evaluating VLM accuracy as standalone agents compared with physician performance reported conflicting evidence. One study assessing VLMs as a clinical copilot demonstrated higher accuracy from the group of physicians using VLM assistance. A meta-analysis could not be performed due to the heterogeneity across study populations and outcomes. The majority of the studies were assessed as having a high risk of bias due to dataset quality. Primary limitations identified across studies include small sample size, a lack of external validation, and the need for prospective clinical deployment studies. No study provided documented considerations regarding model safety or data security. Conclusions: This systematic review suggests that multimodal VLMs consistently outperform unimodal models with access to only image or text. While model performance as standalone agents compared to humans remains inconclusive, a copilot model has demonstrated high diagnostic accuracy. Given substantial methodological concerns across studies, cautious interpretation is required, No firm clinical recommendation can be made regarding the use of standalone VLMs. Further research employing high-quality datasets is needed to ensure the reliability and clinical applicability of future VLMs.
The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus on textual reasoning or isolated visual question-answering (VQA) tasks, lacking holistic integration of clinical narratives and medical imaging, and thus failing to assess the multimodal diagnostic synthesis capability central to expert clinical judgment. To bridge this gap, we introduce MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm. Constructed from case reports sourced from top-tier medical journals and curated clinical case databases, MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes. These cases predominantly represent rare, atypical, or multi-system presentations that demand expert-level evidence integration beyond routine pattern recognition. We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability. Further analysis reveals that medical knowledge proficiency, medical image understanding, and evidence integration are all highly correlated with diagnostic performance.
Lai Wei, Yuchao Chen, Zhenbiao Cao et al.· 0 citations
Vision-language models (VLMs) represent an emerging class of multimodal artificial intelligence (AI) systems that integrate visual information with natural-language understanding and generation. In computational pathology, VLMs provide a framework for aligning histologic morphology from whole slide images (WSIs) with pathology reports, and other text-based knowledge sources. This review summarizes the technical foundations, major applications, evaluation strategies, and deployment considerations of pathology VLMs. Current pathology VLMs support a growing range of use cases, including image-text retrieval, label-efficient classification, visual question answering, abnormality localization, anomaly detection, report generation, and agentic workflow support. These capabilities are enabled by image encoders, text encoders or large language models, multimodal alignment strategies, and, in some systems, generative language components. Despite rapid progress, several barriers remain. Evaluation of pathology VLMs is constrained by limited domain-specific benchmarks, insufficient assessment of visual grounding, overreliance on text-based metrics, vulnerability to hallucination, and uncertain robustness under data shift. Clinical translation also requires validation across institutions, scanners, staining protocols, tissue types, and patient populations, together with workflow integration, regulatory oversight, data privacy, cybersecurity, and pathologist accountability. VLMs are therefore best viewed as assistive systems that may augment rather than replace pathologists. Responsible development will require close collaboration among pathologists, computational scientists, health systems, and regulatory stakeholders to ensure that VLMs improves pathology practice in a safe, interpretable, and clinically meaningful manner.
Rong Xia, Brian R Isett, Jie Chen et al.· American Journal of Patholog...· 0 citations
A definitive taxonomy of the medical VLM landscape is provided, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting-edge frontiers of Dense Pixel-Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning-Incentivized (RL) architectures.
Taha Razzaq, Murtaza Taj, Asim Iqbal· Journal of Biomedical Inform...· 0 citations
This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models have demonstrated significant potential in the field of medical vision-language tasks. However, it is becoming increasingly clear that evaluating these models' performance, whether they are applied to natural or medical images, is challenging. The critical question is whether the models can accurately understand an input image while associating it with relevant input text. To address this, Medical-Checklist imposes a binary test on the models: they are given an image and two captions, where one is correct and the other incorrect, and the model must select the correct one. The incorrect caption contains a single medical concept (word or phrase) that is inaccurately substituted from the correct caption. Although the task is simple, this simplicity enables the unified assessment of diverse multimodal models designed and learned on different principles. It also enables us to verify whether models correctly understand a wide range of medical concepts across various medical sub-domains. Medical-Checklist is designed to reduce potential biases in data and to enable evaluation of the models' ability to handle out-of-distribution inputs, which were difficult in existing datasets. When evaluating four state-of-the-art medical multimodal models with Medical-Checklist, it was revealed that despite their excellent performance in specific tasks such as Med-VQA, they may not correctly understand images, suggesting a long journey ahead for clinical application. The dataset and code will be made public upon acceptance.
Bannapol Limanond, M. Suganuma, Takayuki Okatani· IEEE journal of biomedical a...· 0 citations
Medical large Vision-Language Models (Med-LVLMs) have shown strong potential in multimodal clinical applications such as medical visual question answering and report generation. However, Med-LVLMs remain challenged by hallucinations caused by modality misalignment, where models prioritize textual knowledge over visual evidence and generate outputs that conflict with medical images. To mitigate this issue, recent studies have explored preference optimization to improve image–text alignment, achieving promising results. Despite these advances, existing preference-based meth-ods still face two limitations in medical settings: (1) overfitting to superficial cues, and (2) pseudo convergence of the preference signal. In this paper, we propose Dynamic Evidence-Guided Preference Optimization (DEPO), a new framework that enables evidence-aware and adaptive preference learning for Med-LVLMs. DEPO introduces Multi-Modal Evidence Perturbation (MEP) to suppress non-causal textual and visual shortcuts, and Dispre-ferred Evidence Resampling (DER) to continuously update dispreferred responses as hallucination patterns evolve. Experiments on multiple medical VQA and report generation benchmarks demonstrate consistent improvements over existing methods, with strong robustness across datasets and architectures.
Zixuan Huang, Zhihong Zhu, Xiaolong Liu et al.· Annual Meeting of the Associ...· 0 citations
Oral diseases affect billions of people, yet specialist dental expertise remains unevenly distributed, and diagnosis often requires synthesis across diverse imaging modalities. Existing artificial intelligence systems mostly address isolated tasks, limiting their applicability in comprehensive dental assessment. Here we introduce DentVLM, a dental vision-language model that jointly interprets images and text, supports expert-level oral disease diagnosis across seven dental imaging modalities and 36 tasks. Developed using 110,447 images and 2.46 million bilingual visual question-answer pairs, DentVLM outperforms leading proprietary, open-source and domain-specific medical models on internal and external tests. In a study of 32 participants, DentVLM surpasses junior readers, matches intermediate general practitioners and approaches senior specialists. In collaborative workflows, it raises junior and intermediate readers toward specialist-level performance and reduces diagnostic time for all readers by 15.0-37.0%. These results establish DentVLM as a clinical decision support tool for reducing specialist care gaps and broadening access to high-quality dental expertise. DentVLM is a dental vision-language model developed to support dental diagnosis across seven oral imaging modalities and 36 tasks. It matches intermediate general practitioners, approaches senior specialists, and reduces diagnostic time by 15.0-37.0% in collaborative clinical workflows.
Zijie Meng, Jinxiang Hao, Xi-Wei Dai et al.· Nature Communications· 1 citation