Structure-Level Concept Erasure via Self-Attention Disruption in Diffusion Models

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

Dual-path attention modulation for training-free text-guided image editing

Text-guided image editing with diffusion models faces a core challenge: simultaneously executing semantic edits while preserving source image structure. We propose a training-free Dual-path Attention Modulation (DAR) framework that decouples these objectives. DAR establishes cross-image feature alignment in self-attention for spatial consistency and semantic remapping in cross-attention for precise attribute binding. To overcome limitations of fixed replacement ratios, we introduce Adaptive Self-Attention (ASA) and Adaptive Cross-Attention (ACA) modules that dynamically regulate attention replacement based on real-time visual and semantic feedback. Experiments on Wild-TI2I and ImageNet-R-TI2I demonstrate that our method achieves superior structural fidelity (SSIM: 0.7721 on Wild-TI2I, 33% improvement over PnP) while maintaining competitive semantic alignment, with high efficiency (11.3s per image).

MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers

Text-to-image diffusion transformers learn about objects and scenes by learning to generate them, making them strong candidates for training-free zero-shot open-vocabulary semantic segmentation. State-of-the-art attribution methods score each pixel independently, comparing its features against a fixed text-derived class representation, whether as an output-space similarity or as a cross-attention weight. This discards structured signals the model itself exposes: the temporal structure of the generative trajectory, the visual appearance statistics of each concept, and the image's own pairwise feature geometry. We present MAVISEG, a training-free refinement layer that recovers these signals. Because its operators consume only a pixel-by-concept score field and a pixel feature space, MAVISEG is capture-agnostic rather than tied to one attribution method. Across six benchmarks it achieves the strongest overall results among training-free methods, including the best mIoU on every benchmark. Interestingly, gains are largest where the initial capture is weakest, and individual operators contribute depending on the noise in the field they refine. Our results indicate that diffusion transformers carry more concept-level information than current attribution methods recover, and that much of it is lost on the way to the mask rather than absent from the model.

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models. Drawing inspiration from the concept of"enlightenment"or"aha moment"in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon-a latent capacity for sudden capability boost. Then, we propose Enlightenment, a novel training-free post-tuning paradigm for large-scale models. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training-free ones predominantly manipulate attention weights. We introduce two architecture-specific instantiations: i) For large language models, we propose attention head-mixing shortcuts that recalibrate attention weights by linking the initial attention head's output to all other target heads, modulated by an adaptive scaling factor initialization strategy. ii) For vision-language models, we apply a lightweight scalar-modulated factor to residual connections in the decoder layers, regulating information flow. Extensive experiments show that Enlightenment efficiently unlocks the latent potential of pre-trained networks, yielding remarkable performance improvements across diverse benchmarks and models.

Regional self-attention for layout-to-image diffusion: resolving overlap, scale, and semantic misalignment

Layout-to-image diffusion models have demonstrated significant potential in generating multi-object, position-controllable images. However, this paper identifies three fundamental contradictions in existing methods: difficulty in distinguishing overlapping objects of the same type, background-dominated training leading to the suppression of small objects, and semantic misalignment between generated images and layout annotations. By analyzing the self-attention mechanism in diffusion models, we identify its global receptive field as the root cause of these issues—when the bounding boxes of multiple objects of the same class overlap spatially, the model lacks a mechanism to ensure that pixels within each region focus exclusively on the corresponding object. To address this, we propose the Regional Self-Attention Mechanism, which strictly confines the self-attention operation to the independent regions defined by the target bounding boxes, enabling the model to generate specific-category targets in a targeted manner during the denoising process. Experiments on datasets such as COCO and Visual Genome demonstrate that this method significantly improves the distinguishability of overlapping objects of the same class, the reconstruction accuracy of small objects, and the semantic consistency between layout and image.

CARVE: Concept Avoidance via Reward-shaped Visual Erasure