Open-vocabulary object detection based on fine-grained alignment

Abstract

Open-Vocabulary Object Detection (OVD) aims to break through the category dependency of traditional closed-set detectors and realize object detection driven by arbitrary text labels. However, in complex scenarios, it still faces core problems such as inadequate exploitation of visual semantics, misalignment between regional-category annotations, and weak generalization ability in inference. To address the above problems, this paper proposes an open-vocabulary object detection framework based on fine-grained alignment. Through end-to-end optimization of data construction, model design, and inference procedure, accurate visual-language semantic mapping is achieved. At the data level, a customized Prompt is used to drive the Qwen2.5 VL to generate detailed image-level descriptions. Meanwhile, a "single-region-multi-objects" annotation strategy is proposed to create region-level pseudo-labels, constructing a fine-grained dataset containing 1120k samples. At the model level, a trainable cross-modal projection layer is introduced to uniformly map the image-level features and region-level Queries output by DETR to the Large Language Model (LLM) space. Through the joint training of two tasks—image-level caption generation and region-level phrase generation—the global-local collaborative visual-text alignment is strengthened. At the inference level, Qwen is used to dynamically generate multiple groups of differentiated prompt templates for the target categories, replacing traditional fixed templates and learnable vector templates, thereby improving the generalization ability in open scenarios. Experimental results show that the framework performs excellently on multiple benchmark datasets such as LVIS and COCO-O, demonstrating stronger adaptability to complex scenarios.