A unified latent diffusion framework that addresses both limitations simultaneously, indicating that adaptive modality weighting outperforms fixed conditioning strategies and that modality-dropout training provides an effective partial-conditioning robustness mechanism.
Abstract
Introduction Multi-modal conditioning in latent diffusion models—combining text, structural, and spatial guidance signals—substantially improves controllable image synthesis, yet two limitations persist across most existing frameworks. First, conditioning modalities are fused using fixed architectural weights that remain constant regardless of the input, preventing any input-adaptive rebalancing of guidance. Second, most existing multi-modal approaches are designed under the assumption that all conditioning signals will be present at inference time, leaving the model without any way to degrade gracefully when some modalities are unavailable. This paper introduces a unified latent diffusion framework that addresses both limitations simultaneously. Methods Our architecture integrates three complementary conditioning modalities—semantic text embeddings via CLIP, structural boundary maps derived from Canny edge detection, and regional semantic layout from segmentation masks-as co—equal inputs into a single U-Net denoising backbone. The framework has two core components: Softmax-Normalized Adaptive Guidance Fusion (SNAGF), which replaces fixed fusion weights with three learnable, softmax-normalized modality importance scores optimised end-to-end alongside the diffusion objective; and Modality-Dropout Training (MDT), a structured regularisation strategy that randomly zeroes individual guidance representations with probability pdrop = 0.3 during training, training the model to produce coherent outputs under any available subset of conditioning inputs. Results Evaluated on CIFAR-10, the full SNAGF+MDT framework achieves SSIM 0.93, FID 17.2, PSNR 34.1 dB, CLIP Score 0.88, LPIPS 0.09, and IS 10.4. MDT alone yields an average 15.5% FID improvement across partial-conditioning inference configurations, and learned SNAGF weights converge to λtext = 0.370, λedge = 0.363, λseg = 0.267, demonstrating stable convergence from epoch 30 onward. Results are averaged over three random seeds. Discussion Taken together, they indicate that adaptive modality weighting outperforms fixed conditioning strategies and that modality-dropout training provides an effective partial-conditioning robustness mechanism. All findings are established at 32 × 32 resolution on a single benchmark with algorithmically constructed conditioning signals; validation on higher-resolution datasets with naturally paired text and structural annotation remains necessary before these conclusions can be generalised.
This work introduces a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model, offering a practical latent-variable lens for characterizing continuous multimodal interactions.
Experiments show that TopoSegDiff produces more realistic and structurally faithful image-mask pairs, reduces topology inconsistency, and consistently improves downstream segmentation performance when used for synthetic augmentation, which support topology-preserving conditioning as an effective route toward more relia...
Qin Lei, Ce-Ran Chen, Jiang Zhong et al.· IEEE Transactions on Neural...· 0 citations
Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness, is proposed and suggested as a practical route for robustness correction.
Song-Yuan Sui, Zhen Tan, Mohan Zhang et al.· 0 citations
FuseAdapt is proposed, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation.
The proposed UniDiffFusion framework is trained in a progressive manner to decouple fusion learning, degradation-aware restoration, and application-specific adaptation, thereby reducing interference among heterogeneous objectives.
Xing-Xin Xu, Si-Qi Zhao, Xin Li et al.· 0 citations
Multi-modal image fusion (MMIF) aims to form a single image by integrating shared information, preserving complementary cues, and coordinating cross-modal conflicts across modalities. However, due to the absence of ground-truth fused images, existing MMIF supervision commonly uses spatial-domain sources or gradient var...
Ze-Yu Wang, Ming-Yu Ge, Hai-Yu Song et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.