Jun 2026· arXiv.org· Vol abs/2606.02739· 0 citations· 39 references
Computer ScienceEngineering
TL;DR
EntangleCodec is proposed, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization that achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding, and supports both TTS and TTA generation in a unified framework.
Abstract
Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose \textbf{EntangleCodec}, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream. A flow-matching diffusion decoder further enables high-quality reconstruction across speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to \textbf{+7.4\%} on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio language models demonstrate strong scaling behavior: even at \textit{0.6B} parameters, the model surpasses specialized continuous-representation LLMs with over \textit{13B} parameters across three benchmarks using \textbf{22$\times$} fewer parameters; scaling to \textit{8B} further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec.
Experiments show that BiMTokenizer achieves superior acoustic reconstruction and the lowest WER among low-bitrate codec baselines across both clean and noisy environments, while using less than half the parameters of recent dual-tower baselines, confirming that a well-designed single-tower codec can preserve the semant...
Xin Zhang, Lin Li, Chuan-Bo Liu et al.· 0 citations
This work presents TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU, and is designed primarily for English and German, with additional multilingual support.
MuSeC is proposed, a music semantic codec that factorizes semantic and acoustic content directly from mixed signals without source separation that preserves information required for high-fidelity reconstruction while producing more LM-friendly discrete units.
Hua-Kang Chen, Guo-Bin Ma, Yue-Peng Jiang et al.· 0 citations
This work proposes Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling and is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss.
Pei-Jie Chen, Zhuanling Zha, Zhipeng Nie et al.· 0 citations
TokenMapper is presented, a direction aware framework for direct token to token translation between heterogeneous speech tokenizers in the discrete domain and provides a practical step toward cross model speech token interoperability without intermediate waveform reconstruction.
Tal Kozakov, Tal Rosenwein, Eliya Nachmani· 0 citations
Recent advances in text-to-audio (TTA) and audio-to-audio (ATA) generation models have enabled the creation of highly realistic environmental sounds, raising growing concerns about malicious audio manipulation in real-world scenarios. To address this emerging threat, the ESDD 2026 Challenge was introduced as the first...
Sanghyeok Chung, Seungsang Oh, Donggun Kim et al.· Proceedings of the Thirty-Fi...· 0 citations