Jul 2026· Journal of Artificial Intelligence and Information· 0 citations· 46 references
TL;DR
An overview of Vision LLM architectures, their applications and the challenges they face and case study of how building of AI Models through visionLLM may help IndoAI AI camera system are provided.
Abstract
Vision LLMs are trained on vast datasets containing paired image-text samples, allowing them to perform tasks such as image captioning, visual question answering (VQA) and multimodal reasoning. These Models (Vision LLMs) mark a transformative leap in artificial intelligence by merging visual and linguistic understanding, enabling seamless human-machine communication, power groundbreaking applications-from automated diagnostic reporting in healthcare to real-time scene analysis in autonomous systems. Yet, key challenges remain, including computational inefficiency, embedded biases in training data and limited interpretability which currently restrict broader deployment. Cutting-edge research is tackling these obstacles through optimized model architectures, fairness-aware dataset curation and advanced explainable AI methods. As these advancements progress, Vision LLMs are poised to revolutionize AI- driven solutions across industries such as healthcare, robotics, autonomous vehicles. Their continued evolution is redefining the landscape of interdisciplinary AI, fostering more intuitive, ethical and scalable intelligent systems. This article provides an overview of Vision LLM architectures, their applications and the challenges they face and case study of how building of AI Models through visionLLM may help IndoAI AI camera system.
Detection and localization of AI-tampered images are critical for trustworthy AI, yet modern generative models have made such manipulations increasingly difficult to identify. While traditional binary classifiers can detect image tampering, they lack interpretability and generalization. Vision-Language Models (VLMs) offer a promising alternative due to their strong visual understanding and reasoning capabilities; however, existing approaches typically rely on supervised finetuning with curated explanations rather than exploiting their inherent reasoning capabilities. In this work, we investigate whether VLMs can be trained to reason about AI-generated image edits using reinforcement learning (RL) rather than explicit reasoning supervision. Motivated by the success in Group Relative Policy Optimization (GRPO), an RL technique that incentivizes the model to reason by generating thinking traces prior to giving the final answer, we propose a GRPO-based training framework that utilizes simple accuracy and format rewards. Given an input image, the model produces a structured reasoning trace and predicts whether the image has been tampered with. A lightweight segmentation model is then guided by the reasoning output to generate pixel-level localization masks. Experiments across multiple image manipulation datasets demonstrate that our approach achieves competitive detection and localization performance compared to state-of-the-art image forgery detectors, despite requiring substantially weaker supervision. We introduce effective intersection over union (eff-IoU), a unified metric to jointly evaluate detection and localization. These results suggest that reinforcement learning provides an effective and scalable mechanism for teaching VLMs to reason about AI-generated content.
Darsha Udayanga, Pin-Yu Chen, Payel Das et al.· 0 citations
AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.
Han Bao, Yue Huang, Yanbo Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Driven by multimodal large language models (MLLMs), remote sensing image (RSI) understanding is undergoing a paradigm shift, evolving from learning a domain-specific model to learning a general foundation model with domain adaptation (LaGD). Under the LaGD paradigm, conventional datasets, such as DOTA and RSICD, which fueled progress in RSI understanding over the past decade, are no longer adequate for emerging tasks because their annotation formats are task-specific and lack the language-level supervision required by MLLMs. We argue that a new dataset must be purposefully designed to support three core capabilities: First, generalization, enabling models to learn shared knowledge across tasks through a unified annotation format; Second, complex scene understanding, training models to capture fine-grained object attributes and spatial relationships and to describe scenes in detailed natural language; Finally, reasoning, equipping models with high-level visual reasoning through multiturn dialogues. To this end, we present UniRS-Instruct, a high-quality, diversified, and unified multimodal instruction-following dataset for RSI understanding. UniRS-Instruct unifies diverse tasks, including image captioning, visual question answering, visual grounding, and region-level captioning, into a consistent (question and answer) format. To construct fine-grained and context-aware instruction data, we propose a hierarchical prompting strategy: at the local level, objects are identified via rotated bounding boxes to describe their fine-grained attributes and spatial relationships; at the global level, local information is integrated with the full image to generate detailed scene-level instruction descriptions through GPT-4 V. Extensive experiments on multiple remote sensing benchmarks demonstrate that MLLMs fine-tuned with UniRS-Instruct achieve superior performance in image captioning, visual question answering, and visual grounding tasks, and exhibit stronger capabilities in describing fine-grained information, uncovering implicit knowledge, and conducting complex reasoning compared with models trained on existing datasets.
Linrui Xu, Yuhan Wang, Ling Zhao et al.· IEEE Journal of Selected Top...· 0 citations
This paper presents a multimodal pipeline that combines vision-language captioning models and openvocabulary object detectors to investigate the impact of automatically generated textual prompts on semantic image understanding. The study evaluates several captioning models, including BLIP, BLIP-2, InstructBLIP, and LLaVA, in combination with two open-vocabulary detectors, OWLv2 and Grounding DINO. Experiments conducted on a representative subset of the COCO dataset show that prompt quality significantly influences detection performance and that post-processing operations, including label normalization and filtering, substantially improve semantic detection metrics. The results reveal complementary behaviors between Grounding DINO and OWLv2, highlighting the importance of prompt engineering and output refinement in multimodal vision-language pipelines. Rather than introducing a new detection architecture, this work provides a comparative analysis of the interactions between caption generation, prompt extraction, and open-vocabulary detection, offering insights for the design of future interactive vision-language systems.
Xin Gao, Madjid Maidi, B. Daachi· NLP & Big Data· 0 citations
Vision-language models pretrained on large-scale datasets have become the foundation of cross-modal understanding. However, they often overemphasize salient objects and underperform in compositional reasoning that additionally requires fine-grained perception of attributes and inter-object relations. Although compositional data-driven approaches have shown promise in addressing this challenge, low-quality captions undermine robust alignment and consequently hinder fine-grained perception. Specifically, two fundamental issues arise: (1) low-density positive captions restrict the global representation capability of image encoders, causing local alignment. (2) false-negative captions introduce noise into training, resulting in fragile alignment. In response, we propose the Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP), enhancing both the robustness and granularity of vision-language alignment. To address the first challenge, RoFLIP employs the Densified Ranking Loss (DRL), which promotes global visual representations by contrasting dense and sparse captions. For the second challenge, RoFLIP incorporates Denoised Cross-modal and Text-modal Contrastive Losses (DCCL &DTCL), reducing the impact of noise while effectively leveraging negative captions. In addition, we introduce RoFLIP’s Auxiliary dataset (RoFA), comprising 225K high-quality captions that are fluent and image-related to support training. Quantitative evaluations demonstrate that RoFLIP-CLIP-B/32 achieves an average gain of 7.41%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.41\%$$\end{document} across 5 compositional reasoning benchmarks while maintaining considerable performance on 23 downstream benchmarks. Furthermore, replacing the corresponding modules in LLaVA-v1.5 and Stable Diffusion v1.5 with RoFLIP-CLIP-L/14 also brings performance gains on VQA and image generation tasks, respectively. These results underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuanbin Liu, Shancheng Fang et al.· International Journal of Com...· 0 citations
Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose \textbf{Seg}mentation before \textbf{Answer}ing (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual tokens via positional embeddings. In experiments, we evaluate SegAnswer across diverse benchmarks, including high-resolution perception, general perception, and hallucination. It achieves consistent improvements and also exhibits considerable performance on segmentation tasks, validating its capability for reliable pixel grounding.
Yake Wei, Yuan Wang, Fengyun Rao et al.· 0 citations