Align-TCN: A Hardware-Aligned Streaming Architecture for On-Device Multimedia Concurrency
Abstract
Speech enhancement is an always-on component in mobile multimedia applications, where it must operate under stringent latency budgets alongside other computation-intensive vision tasks, such as real-time HD video calling and concurrent computational photography. Although existing lightweight speech enhancement models deliver strong acoustic performance with minimal parameter counts, their hard real-time streaming deployment on on-device NPU incurs substantial inference latency, which reduces the scheduling budget for concurrent tasks and degrades the overall real-time responsiveness of multimedia systems. To address this issue, we present Align-TCN, a hardware-aligned fully convolutional streaming architecture for on-device deployment. The proposed architecture is co-designed from both representation and computation perspectives, by constructing an aligned spectral representation that better matches NPU execution characteristics and by introducing time-frequency 2D causal convolutions for efficient parallel modeling of streaming speech features. In this way, Align-TCN achieves a better balance between deployment efficiency and enhancement performance under stringent real-time constraints, with a per-frame inference latency of only 1 ms on edge NPU. Extensive experiments on both standalone deployment and end-to-end concurrent multimedia pipelines validate the effectiveness of the proposed design. Compared to existing lightweight baselines, Align-TCN maintains a similar parameter budget while substantially reducing per-frame inference latency on edge NPU. It further improves system-level real-time performance in concurrent multimedia pipelines with 60 FPS guaranteed for concurrent tasks, while preserving competitive acoustic quality.