TriM-D: Accelerating 2-D/3-D Diffusion Generation Via Triple-Mixed Quantization and Sparsity-Aware Dual-Branched Computation

Abstract

Diffusion models have shown marked advancements in 2-D generation and have also become focal points in 3-D generation via consistent multiview image generation. However, the computation and memory demands hinder their real-time deployment on mobile and edge devices. Moreover, the reduction of diffusion timesteps leads to a decrease in interstep similarity, thereby making previous methods ineffective in optimizing computation. The varied layer quantization sensitivity and optimal format in diffusion models also present challenges for traditional quantization methods to achieve efficient memory compression. To address these issues, we first introduce <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula> quantization, a mixed-precision, mixed-format, and mixed-granularity quantization framework that allocates optimal precision and format to different data with a unified FP8 computation flow for efficient implementation. Building upon <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula> quantization, we present TriM-D, a hardware accelerator designed to optimize computation. It features sparsity-aware dual-branched FP8-MAC units that introduce a dual-branch path and an efficient approximate adder to fully utilize bit-level sparsity of <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula>-quantized data. Such an FP8-MAC scheme preserves both efficiency and numerical accuracy. Our experiments demonstrate that TriM-D significantly outperforms the state-of-the-art (SOTA) diffusion accelerator Cambricon-D (Camb-D), achieving an average reduction of 56.5% in memory access and 80.6% in computation cost. In addition, TriM-D provides a <inline-formula> <tex-math notation="LaTeX">$55\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$6.72\times $ </tex-math></inline-formula> improvement in energy efficiency and <inline-formula> <tex-math notation="LaTeX">$3.69\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$2.67\times $ </tex-math></inline-formula> speedups over NVIDIA A100 and Camb-D.