HearFocus: Low-Latency Speech Separation for User-Focused Listening

Abstract

Interactive audio systems must separate overlapping voices quickly enough to preserve conversational timing. We present HearFocus, a causal time-domain separator built on continuous Resonate-and-Fire (RF) dynamics with causally masked attention. A padding-free encoder, cumulative normalization, and causal sequence processing incur approximately 9 ms of nominal algorithmic latency. On a fixed 3,000-mixture LibriSpeech test manifest, HearFocus achieves 9.01 dB SI-SNRi, outperforming a causal Conv-TasNet baseline at 6.53 dB by 2.48 dB (95% CI: 2.39–2.57). On an NVIDIA A40, the final model achieves faster-than-real-time throughput with a median RTF of 0.402. HearFocus is evaluated as a separation backend for user-focused listening; speaker selection remains a downstream interface function.