TORUS: A Unified Closed-Loop Benchmark for Unified Audio Models
Abstract
Unified audio models capable of audio understanding, audio generation and, increasingly, audio editing are proliferating rapidly. However, each capability is evaluated in isolation on specialized benchmarks and a basic question remains unanswered: can these models understand the audio they generate/edit i.e. do they exhibit audio understanding-generation self-coherence (UGC_OWN)? We introduce TORUS, an audio-native unified benchmark for holistic, closed-loop evaluation of unified audio models comprising 1,053 questions (117 tests) spanning 21 tasks and 8 audio domain settings across all 3 audio types. We evaluate unified models and cascades of specialized models across objective and subjective metrics. Best unified models are able to answer only 52.4% questions about their generations, while gpt-audio can answer 53.6% questions. We hypothesize that a truly unified audio intelligence should be highly self-coherent and thus position understanding-generation coherence as an essential test for future unified audio systems. Further details are on our project page: https://torus-benchmark.github.io/
