A Diffusion-Based Data Augmentation Framework for Semantic Segmentation with Limited Data

Abstract

Semantic segmentation relies heavily on large-scale datasets with pixel-level annotations, while dense labeling is costly and difficult to obtain in many practical scenarios. Existing generative data augmentation methods either depend on source segmentation masks, which limits layout diversity, or are insufficient to fully bridge the gap between generative features and discriminative semantic representations, thereby limiting annotation quality in complex scenes. To address this issue, we propose an annotated sample generation framework built on Stable Diffusion 3.5 and a semantic label generator. The generator learns to map intermediate diffusion features to pixel-level semantic labels. During generation, the framework uses Stable Diffusion 3.5 to synthesize images and the semantic label generator to produce the corresponding annotations. Experiments under limited-data settings on Cityscapes, PASCAL VOC 2012, and BDD100K show that the generated samples consistently improve downstream segmentation performance, demonstrating the effectiveness and generalization of the proposed framework.