A prompt extension framework for TUSS is proposed that incorporates downstream task information into the input prompts and switches the loss function according to the given prompt during training, enabling outputs with different signal characteristics at inference time.
Abstract
Task-aware unified source separation (TUSS) enables a single model to handle diverse separation tasks by conditioning on input prompts. However, conventional TUSS does not account for downstream task requirements, such as whether the enhanced speech will be used for human listening or automatic speech recognition (ASR). In this paper, we propose a prompt extension framework for TUSS that incorporates downstream task information into the input prompts and switches the loss function according to the given prompt during training, enabling outputs with different signal characteristics at inference time. Specifically, we introduce an ASR-dedicated prompt paired with a regularized loss function that reduces speech artifacts to improve ASR robustness, while the standard prompt is paired with the conventional SNR loss function. Experiments on the LibriSpeech and JNAS corpora demonstrate that the proposed joint-training scheme enables a single model to improve ASR performance over noisy input across a wide range of SNR conditions by selecting the ASR-dedicated prompt, while maintaining general speech enhancement quality when the standard prompt is used.
Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned featur...
Xiao-Yu Yang, Arthur Hinsvark, Antonios Alexos et al.· arXiv.org· 0 citations
Target speaker extraction (TSE) is a key frontend for enabling audio large language models (LLMs) to recognize and understand a designated speaker in overlapping speech. However, multilingual mixtures exhibit heterogeneous extraction demands, causing fixed-depth systems to either under-process challenging inputs or was...
Yan-Xiang Chen, Hao-Yuan Deng, Jin-Yu Wang et al.· 0 citations
FlowSE integrates flow matching with a Diffusion Transformer (DiT) backbone and delivers promising speech enhancement performance. Its DiT backbone leverages self-attention to capture global context, enhancing generation quality. However, self-attention incurs quadratic complexity w.r.t. sequence length, severely slowi...
Bing-Yu Dai, Xia Zou, Liang Ao et al.· 2026 7th International Confe...· 0 citations
Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the inst...
Hemant Yadav, Sunayana Sitaram, Roger Zimmermann et al.· 0 citations
CRAF, a cross-view residual-aware fusion framework that uses ALLM-guided cross-view attention to enrich SSL representations and adopts ALLM as a high-level reference to separate ALLM-explainable information from complementary SSL residual information, demonstrates robustness to unseen spoofing attacks.
M. Phan, Khalid Zaman, C. Mawalim et al.· arXiv.org· 0 citations
Discrete speech tokens provide a compact interface between speech encoders and large language models for automatic speech recognition, but single-tokenization systems remain sensitive to the chosen encoder. We propose multi-view discrete token augmentation, a simple strategy that augments each training utterance by gen...
Paul Moïse Gangbadja, Mickael Rouvier, Fabrice Lefèvre· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.