Regional self-attention for layout-to-image diffusion: resolving overlap, scale, and semantic misalignment

Abstract

Layout-to-image diffusion models have demonstrated significant potential in generating multi-object, position-controllable images. However, this paper identifies three fundamental contradictions in existing methods: difficulty in distinguishing overlapping objects of the same type, background-dominated training leading to the suppression of small objects, and semantic misalignment between generated images and layout annotations. By analyzing the self-attention mechanism in diffusion models, we identify its global receptive field as the root cause of these issues—when the bounding boxes of multiple objects of the same class overlap spatially, the model lacks a mechanism to ensure that pixels within each region focus exclusively on the corresponding object. To address this, we propose the Regional Self-Attention Mechanism, which strictly confines the self-attention operation to the independent regions defined by the target bounding boxes, enabling the model to generate specific-category targets in a targeted manner during the denoising process. Experiments on datasets such as COCO and Visual Genome demonstrate that this method significantly improves the distinguishability of overlapping objects of the same class, the reconstruction accuracy of small objects, and the semantic consistency between layout and image.