This work proposes SC-Diff, a semantically calibrated latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration, and introduces Semantic-Guided Self-Attention Calibration, which adaptively applies positive biases to query-key pairs of the same category.
Abstract
Visible-to-infrared image translation provides a practical way to expand infrared training data using abundant visible images. Diffusion models are promising for this task because of their strong generative performance. However, existing diffusion-based methods typically use semantic priors only as external conditions, without explicitly regulating token interactions within the denoising network. Consequently, they struggle to preserve object locations, shapes, and semantic layouts required for reliable annotation reuse. We propose SC-Diff, a semantically calibrated latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration. A pretrained SAM3 model with predefined text prompts first extracts category-specific semantic masks from visible images. These masks are merged into a semantic map and fused with the visible image as the input condition. The same map is converted into token-level semantic labels to calibrate self-attention in the denoising network. Based on these labels, we introduce Semantic-Guided Self-Attention Calibration (SGSC), which adaptively applies positive biases to query-key pairs of the same category. The query-wise calibration strength depends on the dispersion of attention across semantic categories and the attention assigned to the query's own category. The original attention scores further modulate the bias, giving greater calibration to same-category keys with stronger responses. This soft calibration reduces cross-category interference while retaining global contextual interactions, thereby improving semantic consistency in generated infrared images. Extensive experiments show that SC-Diff improves perceptual quality and produces more effective synthetic training data for downstream infrared object detection.
SSiffNet is proposed, a diffusion-based framework that explicitly enforces semantic alignment and structural consistency for aerial visible-to-infrared image translation and consistently outperforms existing state-of-the-art methods in both visual quality and quantitative metrics.
Zhe Guo, Lan Wei, Rui Luo et al.· IEEE Transactions on Geoscie...· 0 citations
Results show that SSGLD effectively resolves the trade-off between texture fidelity and pose-variation robustness in high-resolution paired PGPIS under sparse 2D skeleton conditioning.
Aerial visible-to-infrared (V2IR) image translation provides scalable data support for nighttime remote sensing. However, existing methods predominantly learn cross-modal mappings within pixel or generic latent spaces, where physical information acts merely as an auxiliary training constraint. This limitation hinders p...
Leping Lin, Zi-Bin Zheng, Ning Ouyang· IEEE Geoscience and Remote S...· 0 citations
A new ITM framework that improves the model's discriminative performance by focusing on localized core attributes and has a better robustness in handling highly similar hard negatives, which provides a new way to address cross-modal hard sample discrimination.
Ya-Ting Liu, Kuang-Rong Hao· International Conference on...· 0 citations
RGB-thermal image registration is essential for multi-sensor perception, yet accurate alignment remains difficult because RGB and thermal images differ substantially in appearance, gradient structure, and noise characteristics. Even fine-tuning a strong detector-free matcher like LoFTR on cross-modal pairs produces no...
Diffuse2Seg is introduced, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner, and shows that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervise...
Christoph Hümmer, Joachim Sicking, Fabian Hüger et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.