Unified Adaptive Quantization and Representation-Aware Caching for Diffusion Models

Abstract

Diffusion models achieve strong performance in generative modeling but incur high memory and computational costs due to large model parameters and iterative denoising procedures, limiting deployment in resource-constrained environments. This paper presents a unified framework to improving inference latency and memory efficiency of diffusion models by jointly optimizing activation reuse and weight storage. We introduce a representation-aware caching strategy(ReCaT) which reduces redundant computation during inference by selectively reusing intermediate activations along the diffusion trajectory. ReCaT identifies effective caching points based on representational similarity measured using centered kernel alignment(CKA), enabling informed activation reuse without degrading generative quality. To complement activation reuse, we propose Adaptive Clusteringguided Quantization with Entropy Encoding (ACQ-E) for efficient weight storage. ACQ-E uses histogram-guided clustering to model weights, followed by adaptive mixed-precision quantization and entropy encoding to reduce storage and cost. The proposed framework integrates representation-aware caching and adaptive quantization into a unified optimization pipeline, leading to reduced effective bit-width, smaller model size, and lower computational cost while maintaining generative fidelity. Experiments on standard benchmarks, including LSUN Churches, LSUN Bedrooms, CelebA-HQ, MS-COCO, and PartiPrompt, demonstrate efficiency gains with minimal impact on image quality when compared to full-precision.