Diffusion-Based Generative Regularization for Cross-Modal Supervised Learning
TL;DR
Diffusion-based generative regularization is proposed, a supervised discriminative learning framework that leverages a frozen diffusion-based generative model as a regularizer without explicitly generating additional training samples to improve supervised discriminative learning.
Abstract
Ensuring the quality and quantity of labeled training data has been a long-standing challenge in training deep neural networks for discriminative tasks. One solution to this problem is to exploit generative models, but existing approaches often depend on synthetic sample generation, prompt engineering, careful hyperparameter tuning, or modality-specific architectural designs. To address these issues, this paper proposes diffusion-based generative regularization, a supervised discriminative learning framework that leverages a frozen diffusion-based generative model as a regularizer without explicitly generating additional training samples. The framework feeds condition embeddings produced from the discriminative encoder into the diffusion model and jointly optimizes a discriminative loss and a generative regularization loss. We study two regularization losses: noise consistency loss and latent cross-entropy loss. Experiments on image classification with Vision Transformers and Stable Diffusion show improved accuracy on both in-distribution and distribution-shifted benchmarks. We further extend the framework to speech emotion classification by combining self-supervised speech encoders with Grad-TTS, and demonstrate consistent gains on IEMOCAP and RAVDESS under cross-validation. These results indicate that generative regularization provides a modality-agnostic way to improve supervised discriminative learning.