Skip to content

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

Sep 2026 · arXiv.org · Vol abs/2609.25176 · 2 citations · 28 references
Engineering Computer Science

TL;DR

Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate, evaluating audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety.

Abstract

Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $\tau$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.

View source

Similar papers

Preprint Oct 2026

JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction. In this paper, we propose...

Junseok Ahn, Jaehyun Park, Jeongsoo Choi et al. · 0 citations

SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

SALMONN-duo is proposed, an adaptive dual-system voice agent inspired by dual-process theories of cognition that separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM with a powerful asynchronous slow-thinking LLM agent.

Wen-Yi Yu, Si-Yin Wang, T. Chiba et al. · 2 citations
#machine learning Review Oct 2026

Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use

A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and imag...

Aman Tiwari, Akshay Kalkunte, Akintunde Oladipo et al. · 0 citations
Preprint Nov 2026

InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems

Most benchmarks for real-time voice systems focus on one aspect, such as turn-taking, instruction following, paralinguistic control, or conversational memory, each with its own data and set of systems. We present InteractionBench, a benchmark and evaluation harness that measures four dimensions on the same systems from...

Richard Yucheng He, Chen Xu, Yi-Hang Liu et al. · 1 citation

APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction

Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional workflows. We introduce APEX-Voice, a benchmark of 120 interactive professional workflows spanning ten work archetypes such as form completio...

Puneet Mathur, Dinesh Manocha · 2 citations · ⚡1
Sep 2026

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

Real-time full-duplex speech models can listen while speaking, enabling natural interaction without rigid turn boundaries. Existing benchmarks evaluate turn-taking, interruption handling and multi-round dialogue, but largely centre on a designated user rather than an assistant participating in a shared conversation amo...

Chengqian Ma, Wen-Hao Feng, Wei-Xuan Jin et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.