Automatic seed optimization for personalized text-to-image generation with diffusion models

Abstract

Text-to-Image diffusion models are highly effective but remain heavily sensitive to the initial noise. This sensitivity causes significant instability in personalization tasks, where maintaining a specific subject's identity is crucial. While inference-based methods like the W+ Adapter offer efficient alternatives to costly fine-tuning, they suffer from structural conflicts between identity preservation and prompt consistency depending on this noise. In this study, we address this issue by proposing an automated discrete "Latent Space Exploration" framework utilizing random search to optimize seed selection. We compare our discrete seed optimization approach against "Initial Noise Selection," a continuous optimization method that modifies the noise tensor directly via gradient descent. We define a multi-objective scoring function integrating text consistency (CLIP), identity preservation (ArcFace), and structural validity (MTCNN). Quantitative experiments reveal a critical trade-off: while continuous optimization preserves identity competitively, it frequently degrades text consistency by ignoring prompt contexts like clothing or backgrounds. In contrast, our discrete exploration achieves a superior balance, ensuring a 100% face detection rate while maximizing both identity fidelity and text alignment. Furthermore, a subjective evaluation with 151 participants confirms that our method yields significantly higher overall visual quality and prompt fidelity. We conclude that discrete seed optimization offers a robust and practical solution for personalized generation.