Whisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish and provide an evidence base for developing AI medical scribes tailored to Spanish-speaking LatAm.
Abstract
Background
Artificial intelligence (AI) medical scribes rely on speech-to-text (STT) models for transcription. Evaluations of STT models in non-English settings remain scarce. We benchmarked ten STT models on medical consultations from Latin American (LatAm) Spanish and assessed whether fine-tuning improves transcription accuracy.
Methods
Ten YouTube videos depicting medical consultations. Human transcriptions were the ground truth. Five open-source models were evaluated: Whisper Large, Whisper Large v3, Whisper Large v3 Turbo, Voxtral Mini 3B, and Canary 1B v2; and so were five close-source models: gpt-4o-transcribe, gpt-4o-mini-transcribe, gemini-2.5-pro, Eleven Labs, and Assembly AI. Whisper Large v3 was fine-tuned. One video was withheld from training. Performance assessed using Word Error Rate (WER), Character Error Rate (CER), BLEU Score, ROUGE-L, BERT Score, and Semantic Similarity on the one withheld video.
Results
None of the fine-tuning iterations outperformed the vanilla Whisper Large v3. With the withheld video, Gemini-2.5-pro was the close-source model with the best performance in four of six metrics. In comparison to the close-source models, the fine-tuned model never outperformed the other models (withheld video); conversely, in comparison to the close-source models, the fine-tuned model showed better performance across metrics, for instance: BLEU score (63% vs to 58% for the second-ranking model), BERT (89% vs to 86%), and semantic similarity (89% vs to 83%), CER (19% vs 20%).
Conclusions
Whisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish. These findings provide an evidence base for developing AI medical scribes tailored to Spanish-speaking LatAm.
This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis, and shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space.
Souranil Kahali, Rituparna Bose, Abner Hernandez et al.· 0 citations
This study investigates the English-to-Turkish clinical translation performance of three large language models (Aya-23, Seed-X-Instruct, Gemma-3) using zero/few-shot prompting and QLoRA fine-tuning. Our three-stage setup evaluates: (1) general translation after FLORES-devtest training, (2) medical domain adaptation on i2b2 via a mixed dataset, and (3) generalization capacity on the independent FLORES-test set after mixed-data fine-tuning, using SacreBLEU, chrF, and COMET. Results show that Seed-X-Instruct achieves the strongest fine-tuned performance, reaching 32.70 BLEU, 64.34 chrF, and 90.65 COMET on the general FLORES test set, and 22.40 BLEU, 54.61 chrF, and 79.70 COMET on the medical i2b2 set. After mixed-data training, Seed-X improves from 11.48 to 32.39 BLEU on the unseen FLORES test set, whereas Aya-23 drops from 19.23 to 10.83 and Gemma-3 largely preserves its strong zero-shot baseline. These findings indicate that QLoRA effects are model-dependent and that domain adaptation does not yield the same generalization behavior across models.
Muhammetalp Erdem, Ş. Bora, Oguz Dikenelli· Signal Processing and Commun...· 0 citations
BACKGROUND
Large language model (LLM)-powered virtual standardized patients (VSPs) enable scalable clinical skills practice, but the validity of AI-generated scores relative to faculty ratings remains unclear.
OBJECTIVE
To assess agreement between LLM-generated and faculty ratings of history-taking and communication performance, and to examine the influence of rater and case heterogeneity.
METHODS
In this cross-sectional study, 92 fourth-year medical students completed one of three 15-minute voice-based VSP cases (fever, diarrhea, cough). Ten blinded faculty raters scored performance (0-100 total; 0-50 domains). AI scores were generated by DeepSeek-V3 using a calibrated prompt. Agreement was evaluated using mixed-effects models, intraclass correlation coefficients (ICC[2,1]), Spearman correlations, mean absolute error (MAE), Bland-Altman analysis, and variance partition coefficients (VPC).
RESULTS
Median total scores were similar for AI and faculty (93.0 [IQR 6.0] vs 94.0 [IQR 4.0]). Rater variability accounted for 37% of residual variance in faculty total scores (VPC = 0.37). AI total scores were positively associated with faculty total scores (β = 0.37, 95% CI 0.26-0.48, P<.001; Spearman ρ = 0.50, 95% CI 0.34-0.65). Absolute agreement was moderate (ICC[2,1] = 0.51, 95% CI 0.34-0.65), with MAE of 3.11 points. Mixed-effects Bland-Altman analysis showed a non-significant mean bias (1.26 points) and 95% limits of agreement from -4.95 to 7.48 (width = 12.43 points), with proportional bias (β_mean = -0.55, P<.001). Agreement was stronger for information gathering (β = 0.46, ρ = 0.49, ICC = 0.54, VPC = 0.23) than for communication (β = 0.27, ρ = 0.28, ICC = 0.29, VPC = 0.52). A sensitivity analysis in the lowest quartile showed attenuated but consistent agreement (ICC = 0.38).
CONCLUSIONS
LLM-based scoring in a VSP showed moderate agreement with faculty ratings, performing better for information gathering than for communication. Due to rater and case heterogeneity, ceiling effects, and proportional bias, it is suitable for formative use and enhanced sampling in programmatic assessment, but not for independent high-stakes summative decisions.
CLINICALTRIAL
Xiaoxing Gao, Xiaoming Huang, R. Hu et al.· JMIR Medical Education· 0 citations
Automatic speech recognition (ASR), or speech-to-text (STT), is becoming an important interface for AI systems in diagnostic workflows, but general-purpose ASR models often degrade in specialized technical domains. In diagnostic applications such as fault identification, root cause analysis, and repair recommendation, general-purpose ASR systems struggle with domain-specific terminology, abbreviations, part identifiers, and measurement expressions, leading to elevated transcription errors. This work presents a domain adaptation pipeline that unifies three components: a synthetic benchmarking framework in which domain-specific technical text is converted to speech via text-to-speech~(TTS) synthesis and transcribed by open-source ASR models to establish baseline performance; Low-Rank Adaptation~(LoRA)-based fine-tuning of Whisper Large-v3 using those synthetic audio-text pairs; and transfer validation on curated real-world automotive YouTube recordings to assess generalization beyond synthetic conditions. Using automotive technical language as a representative diagnostic domain, a data-scaling study employing progressively larger subsets of in-domain training data evaluates performance on a held-out test set via word error rate~(WER), character error rate~(CER), normalized error metrics, alphanumeric error rate, semantic similarity, and Bidirectional Encoder Representations from Transformers Score~(BERTScore). Results show consistent gains from lightweight domain adaptation on both held-out synthetic data and real-world recordings, confirming that synthetic data generation combined with LoRA-based fine-tuning is an effective and computationally practical strategy for improving ASR accuracy in specialized technical domains where labeled speech is scarce.
Aman Kumar, Ahmed K. Farahat, Huimin Zhuge et al.· PHM Society European Confere...· 0 citations
Background: Speech models (wav2vec 2.0, HuBERT, Whisper), large language models (GPT, LLaMA), and conversational AI have expanded computational speech analysis from handcrafted acoustic features to dialogue-based neurological assessment. How well these approaches address clinical practice has not been evaluated. Methods: We conducted a narrative review searching PubMed, Google Scholar, and IEEE Xplore, supplemented by Interspeech and ICASSP proceedings. Findings are organized along three layers: acoustic-motor (voice quality, prosody, articulation), language-transcript (lexical, syntactic, semantic, and discourse analysis), and integrated multimodal-conversational (interactive dialogue systems). Traditional acoustic biomarkers provide background; the primary focus is on foundation models, LLMs, and conversational AI. Findings: Speech foundation models outperform handcrafted features on several classification tasks but degrade on severely impaired speech due to domain mismatch with healthy training data. LLMs classify transcripts and score cognitive tests, but operate on text alone and cannot access acoustic-motor information. Conversational AI can administer cognitive screening through naturalistic dialogue, but validation is limited to small single-centre feasibility studies. Prospective clinical validation remains limited. Cross-linguistic generalizability is untested for most methods. Interpretation: The field is moving toward integrated speech-language assessment, but the gap between technical capability and clinical utility remains wide. Closing it requires diverse multilingual datasets, standardized benchmarks, prospective validation, and ethical governance.
Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages. However, comprehensive evaluation methodologies that jointly assess perceptual quality, speaker similarity, and acoustic fidelity across diverse speech domains remain limited, particularly for low-resource and underrepresented languages. This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems through domain-specific analysis. The proposed framework integrates complementary subjective and objective evaluation protocols and is demonstrated through a comprehensive case study on a representative low-resource language spanning four speech domains: Formal, Conversational, Literary/Storytelling, and Emotional. Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated using MUSHRA listening tests, ABX discrimination tests, speaker similarity scoring with Resemblyzer, and acoustic analyses based on mel-cepstral distortion (MCD) and F0 RMSE over 960 audio pairs. Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge (mean MCD 12.03 dB; mean F0 RMSE 889 cents), while conversational speech achieves the highest overall acoustic fidelity. Beyond the empirical findings, this work provides a reproducible evaluation framework, publicly releasing evaluation scripts, result tables, and executable Colab notebooks to support standardized benchmarking and future research on TTS evaluation for low-resource languages.
Ali B. Jafar, Amal Sarmad, Shifa Yousaf et al.· 0 citations