Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.
While Chain-of-Thought (CoT) reasoning has improved the capability of language models, directly applying it to Spoken Language Models (SLMs) may introduce long silent intervals under the serial"think-then-speak"paradigm, disrupting real-time spoken interaction. To address this issue, we propose an asynchronous think-al...
Junyi Ao, Kainan Peng, Mingbo Ma et al.· 0 citations
LoopSLM builds on looped Transformers for latent reasoning, reusing a decoder block to refine hidden states with acoustic grounding at every pass, and further narrows the perception-reasoning gap by separating learning to reason from learning to respond, enabling direct inference without CoT.
Sheng-Bo Cai, Yu-Xiang Wang, Jing-Ran Xie et al.· 2 citations
This work first adapts the model to the domain using supervised fine-tuning on synthesized spoken question-answering data and shows that RL improves the accuracy on GSM8K beyond levels previously achieved for speech models only with supplementary reasoning traces, and establishes a new state-of-the-art for mathematical...
Effectively conveying emotional, tonal, and prosodic styles in complex scenarios is essential for text-to-speech (TTS) systems. Current solutions typically follow two paradigms: 1) zero-shot TTS systems deriving voice variability from reference speech prompts, or 2) instructed TTS systems relying on manual textual desc...
Zi-Yue Jiang, Zhou Zhao· Proceedings of the 28th Inte...· 0 citations
Empathetic spoken dialogue is a sophisticated cognitive process that requires not only recognizing emotions but also inferring a user's latent mental states to provide appropriate support. However, current SpeechLLMs often treat empathy as a direct input-to-response mapping, leading to"superficially warm"but emotionall...
Ding-Dong Wang, Shu-Jie Liu, Ya-Yue Deng et al.· 0 citations
Full-duplex spoken dialogue requires simultaneous listening and speaking at sub-second latency, under conversational timing and cognitive demands that change moment to moment. Yet current models mostly impose static operating points, lacking a systematic mechanism for adaptive decisions. We present AdaptDuplex, which e...
Zhi-Yang Zhou, Ying-Xin Shang, Zhou Wang et al.· 1 citation
