Skip to content
Preprint

Downstream-Task-Aware Unified Source Separation

Sep 2026 · 0 citations · 24 references
Engineering

TL;DR

A prompt extension framework for TUSS is proposed that incorporates downstream task information into the input prompts and switches the loss function according to the given prompt during training, enabling outputs with different signal characteristics at inference time.

Abstract

Task-aware unified source separation (TUSS) enables a single model to handle diverse separation tasks by conditioning on input prompts. However, conventional TUSS does not account for downstream task requirements, such as whether the enhanced speech will be used for human listening or automatic speech recognition (ASR). In this paper, we propose a prompt extension framework for TUSS that incorporates downstream task information into the input prompts and switches the loss function according to the given prompt during training, enabling outputs with different signal characteristics at inference time. Specifically, we introduce an ASR-dedicated prompt paired with a regularized loss function that reduces speech artifacts to improve ASR robustness, while the standard prompt is paired with the conventional SNR loss function. Experiments on the LibriSpeech and JNAS corpora demonstrate that the proposed joint-training scheme enables a single model to improve ASR performance over noisy input across a wide range of SNR conditions by selecting the ASR-dedicated prompt, while maintaining general speech enhancement quality when the standard prompt is used.

View source

Similar papers

Sep 2026

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned featur...

Xiao-Yu Yang, Arthur Hinsvark, Antonios Alexos et al. · 0 citations
Preprint Oct 2026

Phonology-Aware Speaker Extraction Frontend for Audio LLMs

Target speaker extraction (TSE) is a key frontend for enabling audio large language models (LLMs) to recognize and understand a designated speaker in overlapping speech. However, multilingual mixtures exhibit heterogeneous extraction demands, causing fixed-depth systems to either under-process challenging inputs or was...

Yan-Xiang Chen, Hao-Yuan Deng, Jin-Yu Wang et al. · 0 citations
Conference Open access Sep 2026

MM-FlowSE: One-Step Speech Enhancement via Flow Matching with State-Space Model

FlowSE integrates flow matching with a Diffusion Transformer (DiT) backbone and delivers promising speech enhancement performance. Its DiT backbone leverages self-attention to capture global context, enhancing generation quality. However, self-attention incurs quadratic complexity w.r.t. sequence length, severely slowi...

Bing-Yu Dai, Xia Zou, Liang Ao et al. · 0 citations
#natural language process... Preprint Oct 2026

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the inst...

Hemant Yadav, Sunayana Sitaram, Roger Zimmermann et al. · 0 citations

CRAF: Cross-View Residual-Aware Fusion for Deepfake Speech Detection

CRAF, a cross-view residual-aware fusion framework that uses ALLM-guided cross-view attention to enrich SSL representations and adopts ALLM as a high-level reference to separate ALLM-explainable information from complementary SSL residual information, demonstrates robustness to unseen spoofing attacks.

M. Phan, Khalid Zaman, C. Mawalim et al. · 0 citations
Sep 2026

Beyond Encoder Fusion: Multi-View Discrete Token Augmentation for LLM-Based ASR

Discrete speech tokens provide a compact interface between speech encoders and large language models for automatic speech recognition, but single-tokenization systems remain sensitive to the chosen encoder. We propose multi-view discrete token augmentation, a simple strategy that augments each training utterance by gen...

Paul Moïse Gangbadja, Mickael Rouvier, Fabrice Lefèvre · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.