Jul 2026· Journal of Biomedical Informatics· pp.
105075
· 0 citations· 48 references
Medicine
TL;DR
A definitive taxonomy of the medical VLM landscape is provided, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting-edge frontiers of Dense Pixel-Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning-Incentivized (RL) architectures.
Abstract
The emergence of foundation models has marked a transformative shift in AI, enabling robust generalization across diverse downstream tasks through putative zero-shot learning. Large Language Models and Vision-Language Models have demonstrated strong capabilities in tasks such as image interpretation, report generation, and question answering by effectively learning from multimodal data-images paired with associated text-often with minimal supervision. In the healthcare domain, this ability to align visual and textual information reduces the reliance on extensive manual annotations, as models can leverage existing clinical reports and imaging data to learn meaningful representations. This integration holds promise for improving diagnostic support, treatment planning, and overall patient care, even in data-constrained settings. In this review, we provide a definitive taxonomy of the medical VLM landscape, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting-edge frontiers of Dense Pixel-Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning-Incentivized (RL) architectures. We critically examine the "medical bottleneck"-identifying the persistent challenges of data scarcity, the "hallucination" risks in generative diagnostics, the computational strain of 3D volumetric processing, and the lack of standardized, clinically-grounded evaluation metrics.
While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.
Zhuoyuan Fu, Zeshang Li, Yiqiong Zhang et al.· 0 citations
A novel multimodal RAG framework tailored for MedVQA is proposed, which leverages multimodal data, including medical images, reports, and generated captions, to provide more accurate clinical answers, and introduces a training paradigm that uses captions as auxiliary supervision, enhancing cross-modal alignment via contrastive learning.
Mai A. Shaaban, M. Zarei, Adnan Khan et al.· 0 citations
Medical Vision-Language Models (Med-VLMs) require reliable reasoning from fine-grained visual evidence, yet existing models can produce plausible clinical answers by relying on language priors or medical templates rather than truly attending to diagnosis-critical regions. On-Policy Distillation (OPD) offers dense token-level supervision on student-generated trajectories and provides a privacy-compatible means of capability transfer without requiring the redistribution of raw patient data. However, standard OPD uniformly distills all tokens, causing sparse evidence-dependent tokens to be diluted by abundant clinical narrative tokens. Inspired by the success of OPD in the large language model community, we propose \textbf{Med-OPD}, to our knowledge the first unified post-training framework that integrates on-policy distillation with medical evidence-aware supervision for Med-VLMs. We introduce \textbf{Medical Evidence Advantage} (MEA), a teacher-grounded counterfactual signal that uses an answer-aware hint to focus teacher scoring on evidence supporting the target diagnosis, and measures each token's dependence on medical visual evidence by comparing teacher likelihoods under the original and evidence-degraded imaging modalities. Based on MEA, Med-OPD redistributes the distillation signal at both the token and trajectory levels, emphasizing diagnosis-critical tokens and evidence-reliant rollouts. Experiments on OmniMedVQA subsets show that Med-OPD consistently outperforms SFT and standard OPD across CT, MRI, Disease Diagnosis, and Lesion Grading. These results demonstrate that evidence-aware distillation can better strengthen medical VLMs'reliance on key visual evidence and improve reliable multimodal medical reasoning. The source code and data is publicly available at: https://github.com/yunhang8658/MedOPD.git
Yunhang Qian, Jiaquan Yu, Jiawei Liu et al.· 1 citation
Advances in multimodal large language models (MLLMs) are extending radiological artificial intelligence (AI) beyond task-specific image analysis toward multimodal understanding and reasoning. Volumetric radiology, however, presents a fundamental representational mismatch: clinical interpretation often requires full-volume spatial context and acquisition-dependent quantitative information, whereas current MLLMs are commonly conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text. Reliable volumetric radiology AI therefore requires representations that preserve task-relevant three-dimensional (3D) information and systems that can access, verify, and integrate this information across clinical workflows. In this Review, we examine more than 200 publications through July 2026. We organize the literature around volumetric representation and multimodal understanding at the model level, agentic orchestration at the system level, and their links to clinical applications and evaluation. We review volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction. We distinguish settings in which selected 2D views or report-mediated reasoning may suffice from those that warrant native volumetric modeling. We also introduce a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation. Across the literature, native volumetric modeling and agentic capabilities depend on the spatial, quantitative, contextual, and workflow requirements of the intended task. Clinical credibility requires faithful volumetric representation, traceable system behavior, claim-aligned validation, and clearly defined human oversight in realistic workflows.
Zanting Ye, Shengyuan Liu, Xin Liu et al.· 0 citations
Medical image segmentation is moving from task-specific convolutional models toward foundation models that can be adapted across organs, modalities, and clinical tasks with fewer manual labels. This transition has been accelerated by self-supervised pretraining, vision-language learning, and promptable segmentation frameworks such as the Segment Anything Model and its medical derivatives. However, the clinical value of these systems cannot be inferred from technical novelty alone. Medical images differ from natural images in dimensionality, intensity statistics, acquisition protocols, disease prevalence, and safety requirements, and recent evaluations show that naive zero-shot transfer remains inconsistent across modalities and lesion types. This narrative review synthesizes literature published up to May 22, 2026, on foundation models for medical image segmentation, with emphasis on technical evolution, application scenarios, validation strategies, and governance needs. Current evidence suggests that foundation models are most promising when they are deployed as interactive, auditable components within human-in-the-loop workflows, where they can reduce annotation burden, support rapid draft segmentation, and improve consistency across large imaging studies. Their translation into routine practice requires external validation, uncertainty-aware quality control, prospective workflow evaluation, bias assessment, and transparent reporting under medical AI guidelines. Future work should prioritize patient-level multimodal modeling, 3D and longitudinal segmentation, federated evaluation, and clinically meaningful endpoints rather than isolated benchmark gains.
Juntao Wei· Scientific Journal of Techno...· 0 citations
Deep learning (DL) and generative artificial intelligence (generative AI) are changing how medical data are analysed and used at the point of care. Our evidence base comprises 80 sources 37 screened studies and 43 landmark primary studies, architectural papers and clinical-AI reporting standards published between 2014 and 2026, across two related domains: medical image analysis and clinical decision support systems (CDSS). We trace the evolution from convolutional neural networks (CNNs) to U-Net and encoder–decoder networks, then to Vision Transformers (ViTs), generative adversarial networks (GANs), diffusion models, large language models (LLMs) and retrieval-augmented generation (RAG). Each model family is compared across nine dimensions: input modality, task, data requirements, validation level, interpretability, failure modes, clinical readiness, regulatory considerations and human-oversight need. Supervised DL reaches clinically useful performance across CT, MRI and pathology on well-scoped detection, segmentation and classification tasks, though results are task-, dataset- and site-dependent and prospective evidence is limited. To address data scarcity, generative models can produce synthetic images or cross-modality translations, but may amplify hidden dataset biases and generate anatomically incorrect images. LLM-based CDSS show promise for guideline-concordant reasoning and medication-safety checks, yet still face hallucination, calibration and regulatory uncertainty. For each technology we provide a deployment-readiness map, review the reporting standards needed for credible clinical evaluation (CONSORT-AI, SPIRIT-AI, TRIPOD + AI, CLAIM, DECIDE-AI, STARD-AI, PROBAST + AI, FUTURE-AI) and set out a research agenda centred on validation, governance and human-in-the-loop deployment. Unlike prior surveys, which treat imaging AI or clinical LLMs separately, we assess both, add an explicit evidence-quality appraisal and link each technology to the reporting standards. On current evidence, we argue that LLM-based CDSS are not supported for routine autonomous use and require clinician supervision. This is not a PRISMA-style systematic review but a structured critical narrative review built on a curated, transparently reported evidence base.
A. Mohan Babu, A. Jeshurun Nehemiah, V. Jagadeep et al.· Frontiers in Digital Health· 0 citations