Toward Efficient Consistency Models via Variance-Reduced Distillation.
TL;DR
This article introduces variance-reduced consistency learning (vrCL), a novel distillation technique that enables stable and efficient training of consistency models without relying on teacher model evaluations, resulting in high computational efficiency and significantly reduced training time.
Abstract
Diffusion models have demonstrated remarkable performance across a wide range of generative tasks; however, their high sampling cost remains a critical bottleneck. To address this, consistency distillation (CD) was proposed, offering a reduction in sampling cost by distilling a pretrained diffusion model. However, achieving generative quality comparable to diffusion models requires extensive training for the distillation process, posing a substantial computational challenge. In this article, we introduce variance-reduced consistency learning (vrCL), a novel distillation technique that enables stable and efficient training of consistency models without relying on teacher model evaluations. By leveraging a student-guided sample pair, vrCL ensures training stability while significantly reducing computational costs. This design eliminates the need for repeated teacher model evaluations during training, resulting in high computational efficiency and significantly reduced training time. Empirical results demonstrate that vrCL achieves competitive generative performance with high training efficiency, reaching strong results within just 100k training iterations.