2026· International Conference on Language Resources and Evaluation· pp. 470-480· 0 citations· 38 references
Computer Science
TL;DR
A linguistically curated bilingual dataset comprising 13,846 utterances distributed across synthetic and literature-derived subsets, with OOV rates reaching up to 100%, and a diagnostic evaluation framework that partitions recognition performance into Biased Word Error Rate (B-WER), which targets domain-specific jargon, and Unbiased Word Error Rate (U-WER), which focuses on general vocabulary is introduced.
Abstract
Evaluating the ability of Automatic Speech Recognition (ASR) models to transcribe specialized vocabulary remains a persistent challenge, as standard datasets predominantly feature common words and thus obscure weaknesses on rare or out-of-vocabulary (OOV) terms. To address this limitation, we introduce a linguistically curated bilingual dataset (English and Portuguese) comprising 13,846 utterances (18.7 hours) distributed across synthetic and literature-derived subsets, with OOV rates reaching up to 100%. We further propose a diagnostic evaluation framework that partitions recognition performance into Biased Word Error Rate (B-WER), which targets domain-specific jargon, and Unbiased Word Error Rate (U-WER), which focuses on general vocabulary. Baseline evaluations with Whisper models ( medium , large-v3 , and large-v3-turbo ) confirm the need for this framework. On the most challenging datasets, B-WER ranges from 0.88 to 0.90, whereas U-WER remains low, ranging from 0.06 to 0.19, demonstrating that conventional WER masks critical failure modes in jargon recognition. Additionally, an oracle upper-bound experiment shows that providing correct jargon via prompting reduces B-WER by 0.50–0.70 absolute, quantifying the considerable potential for contextual biasing. We release the datasets and evaluation scripts as a reproducible benchmark to foster research on domain-aware contextual biasing and OOV handling in ASR systems.
The results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
The results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil et al.· 0 citations
BuzzASR is introduced, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages, which achieve state-of-the-art CER among open-source systems on 27 of 102 languages on the combined FLEURS and Common Voice test set.
Shivam Singh, Aditya Yadavalli, Catherine Arnett et al.· 0 citations
Recognition of domain-specific and low-frequency terms remains challenging for automatic speech recognition (ASR). Although contextual biasing can improve their recognition, directly providing a large terminology dictionary introduces many irrelevant biasing terms. Retrieval-based contextual biasing addresses this issu...
Natsuo Yamashita, Ai Nemoto, Ryosuke Koichi et al.· 0 citations
Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an...
Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency trade-offs are underexplored. We benchmark eleven systems, ten open-weight models and one commercial API, across five Iberian languages (Basque, Catalan, Galician, Por...
Fernando López, Pablo Gómez, David Solans et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.