A Semi-Parametric Framework For Open Vocabulary Panoptic Segmentation
Abstract
Open vocabulary panoptic segmentation aims to jointly predict semantic categories and instance masks for arbitrary userspecified concepts, including categories not observed during training. Existing approaches are predominantly parametric, relying on classifier weights learned from fixed segmentation vocabularies. As a result, their performance deteriorates significantly when transferring to unseen categories and out-of-distribution domains. In this work, we propose SPPS (Semi-Parametric Panoptic Segmentation), a semi-parametric framework that augments parametric segmentation models with an external segment-level visual memory. Specifically, we construct a large-scale memory database of masked segment representations from paired image-text data using open-vocabulary object detection and segmentation. During inference, masked segment features extracted from the query image are used to retrieve semantically similar segment exemplars from the memory. The retrieved neighbors induce a non-parametric classifier via similarity-weighted voting, while a parametric classifier derived from CLIP-based segmentation provides complementary category predictions. The proposed formulation enables scalable open vocabulary segmentation through memory expansion without requiring retraining for new categories. When trained on COCO and evaluated on ADE20k, SPPS achieves 30.9 PQ, 19.3 mAP, and 44.0 mIoU, improving over the parametric baseline by +4.5 PQ, +2.5 mAP, and +10.0 mIoU, respectively.