HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models

摘要

Diffusion models have demonstrated significant applicationsin the field of image generation. However, their high computational andmemory costs pose challenges for deployment. Model quantization hasemerged as a promising solution to reduce storage overhead and acceler-ate inference. Nevertheless, existing quantization methods for diffusionmodels struggle to mitigate outliers in activation matrices during in-ference, leading to substantial performance degradation under low-bitquantization scenarios. To address this, we propose HQ-DM, a novelQuantization-Aware Training framework that applies Single HadamardTransformation to activation matrices. This approach effectively reducesactivation outliers while preserving model performance under quanti-zation. Compared to traditional Double Hadamard Transformation, ourproposed scheme offers distinct advantages by seamlessly supporting INTconvolution operations while preventing the amplification of weight out-liers. For conditional generation on the ImageNet 256×256 dataset usingthe LDM-4 model, our W4A4 and W4A3 quantization schemes improvethe Inception Score by 12.8% and 467.73%, respectively, over the existingstate-of-the-art method.