Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 12550-12560· 0 citations· 17 references
TL;DR
MolGlass is proposed, a data-centric paradigm for visual molecular understanding in vision-language models (VLMs) that injects chemical priors directly into the visual input through chemical-aware visual augmentations, without modifying model architectures or training molecule-specific encoders.
Abstract
Large language models (LLMs) are rapidly becoming interactive scientific assistants. Most existing pipelines rely on specialized molecular encoders and alignment modules, while overlooking the importance of chemical hierarchy and traceable evidence for reliable reasoning and explanation. Thus, we propose MolGlass, a data-centric paradigm for visual molecular understanding in vision-language models (VLMs) that injects chemical priors directly into the visual input through chemical-aware visual augmentations, without modifying model architectures or training molecule-specific encoders. MolGlass enriches depictions with human-interpretable cues such as scaffolds, BRICS-like fragments, and functional groups, with a 2.5D visual encoding strategy that strengthens stereochemical symbols in 2D diagrams to mitigate the gap between 2D and 3D reasoning. Moreover, MolGlass includes a three-stage curriculum (Perceive-Reason-Converse) that progressively trains models from grounding MolGlass conventions to evidence-based reasoning and finally to multi-turn, assistant-style molecular consultation. Across molecular understanding and property prediction tasks, extensive experiments show that MolGlass improves transfer and zero-shot performance and enables visually grounded explanations by tracing model decisions to human-identifiable regions. Our code is available at: https://github.com/RunqingXuCn/MolGlass.
Spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world. Many spatial tasks are grounded in continuous visual scenes, where locations, regions, and paths are more naturally expressed by pointing, marking, or drawing than by reporting precise coordinates or discrete textual symbols. Yet existing spatial reasoning benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch for image-generation models. This makes it difficult to evaluate image-generation models under the same task semantics as text-output VLMs, despite their ability to externalize spatial judgments directly in pixel space. We propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. ProVisE also includes an Agentic builder that constructs and validates task-specific protocols for new benchmarks. We further introduce SpatialGen-Bench, a curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. We evaluate representative text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks. Results show that image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. These findings reveal complementary strengths of pixel-space expression and text-based reasoning and establish a metric-compatible testbed for studying spatial cognition in image-generation models.
Xu Wang, Kaixiang Yao, Miao Pan et al.· 1 citation
Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.
Changjiang Jiang, Qiannian Zhao, Lei Xin et al.· 0 citations
Evaluating representative proprietary and open-source multimodal models, it is found that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks.
Siyu Yan, Zhuoran Yan, Haiying Xu et al.· 0 citations
Vision-Language Models (VLMs) have achieved remarkable progress in aligning visual and textual information; however, their inference processes remain largely implicit, end-to-end, and weakly structured. As a result, even state-of-the-art models often struggle with logical consistency, spatial reasoning, multi-entity binding, and robustness to occlusion or viewpoint changes, limiting their reliability in scientific, industrial, and safety-critical applications. To address these limitations, we propose Structure-Aware Visual Reasoning (SAVR), a conceptual framework that augments VLMs with explicit representation and constraint-based reasoning components. SAVR decomposes visual inference into three interpretable stages: (i) an entity–attribute–relation (EAR) representation, (ii) constraint-aware reasoning over structured representations, and (iii) integration of structured predictions with VLM-generated outputs. This formulation treats visual reasoning as an explicit process that enforces spatial, logical, and physical constraints over grounded entities, thereby improving consistency, transparency, and controllability. Furthermore, SAVR unifies several previously fragmented research directions-including scene graphs, object-centric learning, neuro-symbolic reasoning, and spatially grounded VLMs-into a coherent architectural perspective. Through this synthesis, we clarify how explicit structural representations can systematically mitigate common failure modes of end-to-end VLMs. While this work is conceptual and does not include empirical evaluation, it provides rigorous problem formulation, design rationale, and comparative analysis that establish a foundation for future structure-aware multimodal reasoning systems.
Document question answering over scanned pages requires two coupled abilities: (i) canonicalizing complex layouts into a faithful textual structure, and (ii) selecting and reasoning over query-relevant evidence from that structure. Most existing pipelines decouple OCR from retrieval-augmented reasoning and optimize OCR for global reconstruction, which often misaligns with evidence needs and causes brittle grounding in multi-page settings. We propose UniDocVLM, an end-to-end framework that unifies OCR and visual RAG within a single vision-language model: the model first generates a structured parse of retrieved pages, then activates question-relevant evidence from the parse to support grounded reasoning and answering. To train UniDocVLM under heterogeneous supervision, we introduce a unified JR-GRPO reinforcement learning recipe with lightweight, verifiable rewards, including format, layout-aware OCR, evidence-consistency, and answer-correctness signals, and route them to the corresponding parts of the output to improve credit assignment and reduce interference. Experiments on multi-page document QA benchmarks show that UniDocVLM yields more reliable evidence grounding and improves downstream accuracy under complex layouts.
Zongsheng Cao, Anran Liu, Jun Xie et al.· Proceedings of the 32nd ACM...· 0 citations
MolLingua, a token-centric, dual-modal framework designed for native molecular understanding, uses a dual-branch Residual Vector Quantization engine to discretize heterogeneous, high-dimensional spatial 2D and 3D features into compact code sequences rather than relying solely on continuous projections.
Haoyang Liu, Xikang Feng, Fei Guo et al.· IEEE journal of biomedical a...· 0 citations