HPQ: A Hybrid Framework for Joint Pruning and Quantization of Self-Supervised Speech Models

Abstract

Despite achieving state-of-the-art accuracy in speaker verification (SV), large-scale self-supervised learning (SSL) speech models remain difficult to deploy on edge devices because of their computational and memory demands. Existing compression approaches improve efficiency through pruning and quantization, but usually optimize them sequentially and thus overlook their interaction. In this paper, we propose HPQ, a framework that jointly optimizes differentiable structured pruning and learnable quantization in a single fine-tuning stage. By integrating <inline-formula><tex-math notation="LaTeX">$L_{0}$</tex-math></inline-formula> regularization with Learned Step Size Quantization (LSQ) into a unified objective, HPQ enables the network to co-adapt its architecture to quantization noise. Experiments on VoxCeleb demonstrate that HPQ establishes a new Pareto frontier: an 8-bit, 70% sparse WavLM model achieves a <inline-formula><tex-math notation="LaTeX">$13\times$</tex-math></inline-formula> reduction in model size and a <inline-formula><tex-math notation="LaTeX">$15\times$</tex-math></inline-formula> reduction in bit-operations, with only a 0.23% absolute EER degradation compared with the full-precision baseline. We further show that larger backbones and more diverse training data improve robustness under aggressive compression across both WavLM and W2V-BERT.