Skip to content

OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

Mar 2026 · arXiv.org · Vol abs/2603.23938 · 1 citation · 51 references
Computer Science

TL;DR

OmniACBench, a benchmark for evaluating context-grounded acoustic control in omni-modal models, is introduced and three common failure modes are identified-weak direct control, failed implicit inference, and failed multimodal grounding-providing insights for developing models that can verbalize responses effectively.

Abstract

Most testbeds for omni-modal models assess multimodal understanding via textual outputs, leaving it unclear whether these models can properly speak their answers. To study this, we introduce OmniACBench, a benchmark for evaluating context-grounded acoustic control in omni-modal models. Given a spoken instruction, a text script, and an image, a model must read the script aloud with an appropriate tone and manner. OmniACBench comprises 3,559 verified instances covering six acoustic features: speech rate, phonation, pronunciation, emotion, global accent, and timbre. Extensive experiments on eight models reveal their limitations in the proposed setting, despite their strong performance on prior textual-output evaluations. Our analyses show that the main bottleneck lies not in processing individual modalities, but in integrating multimodal context for faithful speech generation. Moreover, we identify three common failure modes-weak direct control, failed implicit inference, and failed multimodal grounding-providing insights for developing models that can verbalize responses effectively.

View source

Similar papers

Preprint Sep 2026

Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

Recent Omni-Modal Generative Models (Omni-Models) have advanced content generation toward unified modeling of text, images, video, and audio. MiniMax-H3 exemplifies this transition by combining multimodal context understanding with joint audio-visual generation in a shared latent framework. Its unified architecture rai...

Hao-Yu Zhao, Zi-Hao Zhao, Tian-Yuan Deng et al. · 1 citation
#artificial intelligence Preprint Aug 2026

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce \textbf{Ex-Omni-2D}, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue m...

Haoyu Zhang, Zhipeng Li, Xiaoying Tang et al. · 0 citations
Preprint Sep 2026

When Capabilities Fail to Compose: Diagnosing the Compositionality Gap in Large Audio-Language Models

Large audio-language models (LALMs) perform strongly on individual audio tasks, but whether these capabilities can be reliably composed remains underexplored. We conduct a controlled diagnostic study of capability composition in LALMs, requiring models to integrate audio-attribute recognition, cue-conditioned segment s...

Chien-Feng Liu, Chih-Kai Yang, Bo-Han Feng et al. · 0 citations

CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification

CoLMbo-SV, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports, substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an e...

Massa Baali, Sarthak Bisht, Zi-Yue Qiu et al. · 0 citations
Preprint Sep 2026

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues and OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat.

Haolin He, Yun-Fei Chu, Qi Chen et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.