MM-FlowSE: One-Step Speech Enhancement via Flow Matching with State-Space Model
Abstract
FlowSE integrates flow matching with a Diffusion Transformer (DiT) backbone and delivers promising speech enhancement performance. Its DiT backbone leverages self-attention to capture global context, enhancing generation quality. However, self-attention incurs quadratic complexity w.r.t. sequence length, severely slowing inference for long speech signals, while the large parameter count hinders deployment on resource-constrained devices. To address these bottlenecks, we propose MM-FlowSE, a lightweight reformulation of FlowSE from two complementary perspectives. At the training-objective level, we adopt MeanFlow to reformulate multi-step flow matching as single-step average velocity prediction, reducing the number of function evaluations to one. At the architecture level, we replace the heavy DiT backbone with Mamba, a state-space model that achieves linear complexity while efficiently capturing long-range dependencies. These two modifications are orthogonal yet jointly contribute to model compactness. Experiments on VoiceBank+DEMAND show that MM-FlowSE maintains enhancement quality comparable to FlowSE, while cutting parameter count by over 80% and accelerating inference by over 4 times. Our dual-level reformulation thus offers a practical and efficient solution for real-time speech enhancement under resource constraints.