Auditory-inspired binaural speech enhancement using vision transformer with binaural cues

Abstract

Binaural speech enhancement plays a critical role in modern audio technologies, including hearing aids, augmented reality systems, and intelligent acoustic interfaces, where the objective extends beyond noise suppression to preserving spatial cues essential for natural auditory perception. Real-world acoustic environments often contain highly non-stationary noise and reverberation, posing substantial challenges for both classical signal processing methods and contemporary neural network approaches, frequently resulting in distortions of interaural time difference (ITD), its frequency-domain counterpart, interaural phase difference (IPD), and interaural level difference (ILD) cues. Such distortions are particularly detrimental for hearing-impaired listeners, who rely heavily on spatial cues to segregate target speech from competing noise. To address these limitations, this thesis proposes BinauralViT, a transformer architecture for binaural speech enhancement that integrates biologically inspired attention with top-down modulation, representation integration, and a transformer-based layer for capturing temporal coherence. Experimental results on binaural datasets demonstrate that BinauralViT achieves improved speech quality and superior preservation of binaural spatial cues under unseen and non-stationary noise conditions, as evidenced by lower IPD and ILD errors compared with baseline systems. The model also maintains competitive speech intelligibility and effective noise reduction, indicating that the proposed framework enhances speech quality while preserving spatial information critical for realistic binaural perception.