Jul 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP, pp. 1-18· 0 citations
Medicine
TL;DR
A FMs based SGG (FMSGG) framework is proposed, leveraging the broad pretraining of FMs to benefit tail relationships and zero-shot triplets, and introduces the triplet-aware logit adjustment method to mitigate long-tail bias.
Abstract
Scene Graph Generation (SGG) is an essential task at the interface of computer vision and natural language processing, aimed at creating structured semantic representations of visual scenes. However, traditional paradigms that train from scratch with SGG datasets face persistent issues, including model bias due to long-tail distributions and difficulties with zero-shot triplet recognition due to sparse annotations. Recently, Foundation Models (FMs), pretrained on extensive datasets, have demonstrated remarkable success in various downstream tasks, showcasing their potential in enhancing SGG task. Consequently, we propose a FMs based SGG (FMSGG) framework, leveraging the broad pretraining of FMs to benefit tail relationships and zero-shot triplets. FMSGG addresses three key challenges when incorporating FMs into SGG task: inefficient training and inference, difficulty in distinguishing similar relationships, and dual-bias. Specifically, our method involves isolating relationship embeddings from image features obtained from FMs via relationship prompts, and augmenting text prompts to enhance the distinctiveness of relationship description embeddings. Additionally, we introduce the triplet-aware logit adjustment method to mitigate long-tail bias, and propose adaptive evidential smoothing to specifically alleviate inherent biases in FMs. Experimental results across various evaluation modes and metrics indicate that FMSGG achieves state-of-the-art mean recall rate and zero-shot recall rate, while also providing an optimal balance between mean recall rate and recall rate.
The Cross-Modal Dual-Path Generator (CM-DPG), a model for robust open-vocabulary SGG that enhances object-level semantic representations through joint visual-textual encoding and improves relational reasoning using complementary visual and geometric cues, is proposed.
Ming-Hao Zou, Qingtian Zeng, Shangkun Liu et al.· 0 citations
The development of Embodied AI urgently necessitates high-fidelity environment modeling enriched with spatial context. However, existing 3D semantic scene understanding methods predominantly focus on isolated instance-level labels or high-dimensional semantic vector embeddings, lacking effective modeling of explicit spatial-semantic relationships between objects within a scene. To address this issue, we use a method that guides Vision-Language Models(VLMs) to learn scene spatial layout relationships via Supervised Fine-Tuning (SFT). First, based on the InteriorGS dataset, we construct a Visual Question Answering (VQA) dataset spanning 100 scenes, comprising RGB images and grounding images with 2D bounding box prompts. Within this dataset, we systematically annotate the spatial-semantic relationship graphs among visible instances. Second, we utilize the parameter-efficient fine-tuning strategy of Low-Rank Adaptation (LoRA) to enhance the spatial relationship reasoning capabilities of the baseline model, Qwen2.5-VL-7B-Instruct. Furthermore, we design an easy-to-hard, three-stage curriculum learning scheme: progressing from single-image single-instance relationship reasoning, advancing to single-image multi-instance relationship understanding, and ultimately achieving global spatial layout perception across continuous frames. Comprehensive evaluations demonstrate that our method enables the model to effectively comprehend and output spatial-semantic relationship triplets in a predefined format, significantly outperforming the baseline model in inter-instance spatial-semantic reasoning. Our research validates the feasibility of endowing existing VLMs with preliminary spatial intelligence via SFT, laying the foundation for constructing next-generation semantic scene representations enriched with spatial relational information.
Zhe Zhong, Qin-Yuan Ren· 2026 IEEE International Conf...· 0 citations
A 3D Local- Global Linear Attention Mechanism (LG-LAM) is devised that efficiently captures long-range contextual information with linear complexity, enabling a comprehensive understanding of the 3D scene without heavy computational burdens.
Jie Li, Jiaheng Xu, Laiyan Ding et al.· International Conference on...· 0 citations
Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as $\langle$subject, predicate, object$\rangle$ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@$K$ scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.
John Helsby, Yi Yang, B. Rosenhahn et al.· 0 citations
Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).
Harsha Patnala, Debopriyo Banerjee, A. Munot et al.· 0 citations
This work proposes AMCA, an unbiased SGG framework integrating adaptive multi-prototype learning with cross-modal alignment, which achieves consistently competitive performance across multiple SGG tasks, with particularly strong improvements on the unbiased mR@K metric.
Jinhao Fan, Yuanhao Xi, Chuanping Hu et al.· Journal of King Saud Univers...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
