Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play

Abstract

Evaluating spoken role-play requires judging not only linguistic content but also prosody, emotional expression, character embodiment, and contextual fit. Existing automatic evaluation methods often collapse these aspects into coarse scores and are evaluated primarily on synthesized speech. We introduce Speech-DRAME, a framework comprising Speech-DRAME-EvalBench, a bilingual, human-annotated benchmark with contrastive role-, context-, and delivery-mismatch conditions, and DRAME-Eval, a fine-tuned audio-language evaluator. The benchmark supports two complementary protocols: top-down archetype evaluation and bottom-up realism evaluation, with predominantly synthesized benchmark-distribution data and a held-out stress test consisting entirely of authentic human acting. On the benchmark-distribution sets, DRAME-Eval achieves pooled Pearson correlations of 0.629 for archetype evaluation and 0.602 for realism evaluation, significantly outperforming the evaluated zero- and few-shot audio-language judges; modality controls show that its judgments depend on both speech and textual context. This alignment does not transfer to the authentic-speech test set, where DRAME-Eval obtains an average correlation of -0.142 and a within-speaker correlation of +0.029, compared with matched human single-rater agreement of +0.34. These results identify a substantial synthetic-to-authentic transfer gap and establish Speech-DRAME as a resource for studying and improving evaluation of acting differences among real human performances.