CCD: Continual Consistency Diffusion for Lifelong Generative Modeling

Abstract

While diffusion-based models have shown remarkable generative capabilities in static settings, their extension to continual learning (CL) scenarios remains fundamentally constrained by Generative Catastrophic Forgetting (GCF). We observe that even with a rehearsal buffer, new generative skills often overwrite previous ones, degrading performance on earlier tasks. Although some initial efforts have explored this space, most rely on heuristics borrowed from continual classification methods or use trained diffusion models as ad hoc replay generators, lacking a principled, unified solution to mitigating GCF and often conducting experiments under fragmented and inconsistent settings. To address this gap, we introduce the Continual Diffusion Generation (CDG), a structured pipeline that redefines how diffusion models are implemented under CL and enables systematic evaluation of GCF. Beyond the empirical pipeline, we propose the first theoretical foundation for CDG, grounded in a cross-task analysis of diffusion-specific generative dynamics. Our theoretical investigation identifies three fundamental consistency principles essential for preserving knowledge in the rehearsal buffer over time: inter-task knowledge consistency, unconditional knowledge consistency, and prior knowledge consistency. Motivated by these insights, we further propose Continual Consistency Diffusion (CCD), a principled training framework that enforces these consistency objectives via hierarchical loss functions: <inline-formula> <tex-math notation="LaTeX">${\mathcal {L}}_{IKC}$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">${\mathcal {L}}_{UKC}$ </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">${\mathcal {L}}_{PKC}$ </tex-math></inline-formula>. This framework fosters long-term retention of generative knowledge and stable integration of new capabilities. Experiments show that CCD achieves state-of-the-art continual performance across various benchmarks, especially improving continual generative metrics in overlapping-task scenarios.