Dictionary-Augmented Large Language Model Postprocessing for Bilingual Code-Switched Medical Speech Recognition: Development and Evaluation Study

Abstract

Abstract Background Clinical documentation burden contributes significantly to physician burnout, with health care professionals spending much of their time on electronic health record interactions. Automatic speech recognition (ASR) systems offer a promising solution; however, their application in Korean medical settings faces unique challenges due to widespread Korean-English code-switching, where clinicians routinely alternate between Korean conversational language and English medical terminology within single utterances. Objective This study aimed to develop and evaluate a hybrid postprocessing approach combining medical terminology dictionary normalization with large language model (LLM)–based postprocessing to improve ASR accuracy for Korean-English code-switched medical speech. Methods We constructed a speech dataset from 23,652 nursing progress notes, with a linguistic composition of 67.73% (512,626/756,866) Korean, 23.54% (178,166/756,866) English, and 8.73% (66,074/756,866) numerals or special symbols. Four Korean nurses recorded the notes using 5 microphone types in an acoustically isolated environment. Speech recognition was performed using OpenAI’s gpt-4o-transcribe model. For postprocessing, a medical terminology dictionary containing 1070 mapping entries was constructed from 1000 nursing progress notes to normalize Korean phonetic renderings of English medical terms. Six LLMs (2 GPT and 4 Claude variants) were then evaluated across 5 temperature settings (0.0‐0.8). Performance was assessed using BERTScore (bidirectional encoder representations from transformers score; F1), Sentence-BERT cosine similarity, word error rate, and character error rate (CER), comparing postprocessed outputs against the original written notes. Statistical significance was assessed using paired Wilcoxon signed-rank tests with Holm correction (α=.05). Results Temperature optimization showed that all postprocessing models had small temperature-related effect sizes (all |Cohen dz| ≤0.15), with GPT-4o exhibiting the largest dependency and statistically significant improvement at temperature 0.6 (Holm-adjusted P<.001 for both BERTScore and CER) and the 4 Claude variants and GPT-4.1 exhibiting practically consistent performance across all settings. Baseline ASR achieved a BERTScore of 0.9131 and CER of 0.2336. Dictionary-based normalization performed 43,507 word-level substitutions in 70.8% (16,754/23,652) of transcribed sentences. LLM-only postprocessing reduced CER by 36.09% (Claude Sonnet 4) and 32.53% (GPT-4o) compared to baseline. The combined dictionary-LLM approach achieved the best performance: Claude Sonnet 4 attained a BERTScore of 0.9638 and CER of 0.0820, representing a 64.9% reduction in CER from baseline (P<.001). Conclusions The hybrid pipeline integrating rule-based dictionary normalization with LLM postprocessing significantly improved Korean-English code-switched medical ASR accuracy. Dictionary-based normalization yielded consistent additional improvements over LLM-only postprocessing for both GPT-4o and Claude Sonnet 4. As this modular framework requires no model retraining, it offers a practical means of mitigating multilingual challenges in medical ASR.