Language Interpreter: Evaluating Speech Recognition Errors in Speech to Text Translation
TL;DR
This study presents a reproducible synthetic benchmark for evaluating error propagation in a speech-to-text pipeline that links acoustic condition, transcription-error type, deterministic normalization, phrase-aware mapping, sentence correctness, and semantic preservation on the same records.
Abstract
Communication between hearing individuals and people who depend on sign language or on its speech to text translation for understanding what the other person is speaking; remains difficult in many everyday situations, particularly when a trained interpreter is unavailable. This study presents a reproducible synthetic benchmark for evaluating error propagation in a speech-to-text pipeline. This integrates controlled synthetic speech, deterministic text normalization, phrase-aware mapping, and visual-sign reference scoring. The supplied dataset contains 300 reproducible WAV files generated from 15 synthetic voice profiles and 20 fixed sentences per profile, with 150 Quiet and 150 Moderate-noise files. All audio is 16 kHz, 16-bit mono WAV with duration, target/measured SNR, synthesis-profile metadata, and SHA-256 provenance. OpenAI Whisper small multilingual is retained as the architectural reference ASR model because its published configuration provides a realistic deployment target. The present study, however, deliberately evaluates a controlled algorithmic ASR-error layer rather than claiming executed Whisper performance. The workbook contains 300 simulated ASR-like transcripts, 300 resource-anchored benchmark ISL gloss references, 300 phrase-first/lexical-fallback outputs, and 300 direct lexical baseline outputs. The Sentence_Codebook contains 20 fixed sentences and 49 unique normalized lexical tokens. The mean simulated WER was 9.41% (95% bootstrap CI 7.39-11.56), with 3.50% in Quiet and 15.31% in Moderate noise. Overall token-level benchmark sign accuracy was 90.83% (95% bootstrap CI 88.50-93.00), sentence-level benchmark accuracy was 76.00% (95% Wilson CI70.86-80.48), and semantic end-to-end success was 90.00% (95% Wilson CI 86.08-92.91). The direct lexical baseline achieved 14.00% sentence-level accuracy versus 76.00% for the phrase-aware mapper, a paired improvement of 62.00 percentage points (95% paired-bootstrap CI 56.33-67.33; McNemar exact p<0.001). These values characterize the synthetic benchmark and are not presented as human-speech, production-ASR, or expert-adjudicated natural-ISL performance. Beyond assembling existing components, the scientific contribution is an auditable error-propagation benchmark that links acoustic condition, transcription-error type, deterministic normalization, phrase-aware mapping, sentence correctness, and semantic preservation on the same records. An ablation separates direct lexical lookup (14.00% sentence accuracy), exact phrase retrieval (73.67%), and phrase-first mapping with lexical fallback (76.00%). The study is therefore positioned as a reproducible simulation, with external literature used for contextual triangulation.