This paper investigates the effectiveness of pre-trained models like Wav2Vec XLSR-53 and Whisper-Small for developing ASR systems for the Telugu language, addressing the challenge of limited data availability and demonstrating satisfactory results even when fine-tuned on a smaller dataset.
Abstract
This study presents the development of an automatic speech recognition (ASR) system tailored for Telugu, one of the widely spoken Indian languages. In recent years, deep learning (DL) techniques have been applied to develop ASR systems across various languages and domains. These models, however, require substantial training resources and extensive corpora of continuous speech composed from multiple dialectal speakers, along with their corresponding transcripts. This paper investigates the effectiveness of pre-trained models like Wav2Vec XLSR-53 and Whisper-Small for developing ASR systems for the Telugu language, addressing the challenge of limited data availability and demonstrating satisfactory results even when fine-tuned on a smaller dataset. We utilized approximately 20 h of speech data comprising 17,421 sentences of the Telugu language. The models are fine-tuned on four publicly available datasets, including OpenSLR, Common Voice, IndicVoices, and IndicTTS, to introduce greater diversity in both speaker demographics and linguistic content. The Wav2Vec XLSR-53 model achieved a word error rate (WER) of 27.3% and a character error rate (CER) of 6.8% on the test dataset, whereas the Whisper-Small attained a WER of 28.67% and a CER of 7.55%. In addition, performance of the models was evaluated by introducing noise to both individual datasets as well as a combined noise dataset. The results show that, on the combined noise dataset, Wav2Vec XLSR-53 achieved a WER of 19.59% and a CER of 4.58%, while Whisper Small obtained a lower WER of 13.97% and a CER of 3.54%. These results underscore the usefulness of leveraging pre-trained architectures in low-resource linguistic scenarios such as Telugu.
Online voice-based applications and speech communication have grown as a result of the revolutionary rise of smart gadgets and social media. The rapid advancement of deep learning (DL) has transformed the field of audio processing, enabling smooth human-computer interaction. DL approaches have been used to develop speech-to-text (STT) systems across various languages and topics. These models require a large amount of training data: extensive corpora of continuous speech utterances collected from numerous speakers, along with their corresponding transcripts. In this study, we explore the use of state-of-the-art pre-trained models like Wav2Vec2 XLSR-53 and Whisper-small for developing STT systems in the Telugu language, addressing the challenge of limited data availability and demonstrating satisfactory results [23.62% Word Error Rate (WER), 4.12% Character Error Rate (CER)] even when fine-tuned on a smaller dataset. To evaluate model performance, we employed a k-fold cross-validation approach with values of
k
= 2 to
k
= 5, and compared the results with the conventional train-test split method. The results indicate that at
k
= 5, the Wav2Vec2 XLSR-53 model achieved a cross-validation Character Error Rate(WER) of 23.62% and a Character Error Rate (CER) of 4.12%, and the Whisper-small model yielded a cross-validation WER of 28.67% and a CER of 5.48%. These findings suggest that the k-fold cross-validation strategy, at k = 5, enhances the robustness of Wav2Vec2 XLSR-53 in low-resource language scenarios where training data is limited. For Whisper-small, however, the baseline train-test split (WER: 27.76%) outperformed all k-fold configurations tested (k = 2: 32.05%, k = 3: 29.72%, k = 4: 29.40%, k = 5: 28.67%), indicating that the benefit of k-fold cross-validation observed for Wav2Vec2 XLSR-53 does not generalize across model architectures. Additionally, the models were evaluated on unseen noisy data, and both models demonstrated satisfactory performance.
Muzaffar Ahmad Dar, Sri Gani Kaarthikeya Kammula, J. Pushparaj et al.· Frontiers in Artificial Inte...· 0 citations
Findings indicate that language-specific fine-tuning plays a more critical role than multilingual generalization in achieving accurate ASR for Indonesian and provide practical guidance for deploying ASR systems in low-resource language scenarios.
J. Hebert, Amalia Zahra· Bulletin of Electrical Engin...· 0 citations
A preliminary study on the adaptation of Whisper for Automatic Speech Recognition in Baniwa, an indigenous Arawakan language spoken in Brazil, Colombia, and Venezuela, demonstrating that multilingual foundation models can be successfully adapted to extremely low-resource indigenous languages.
Leonardo Duart, T. Fonseca, T. Chacon· 0 citations
SraVaani-1.0 achieves the lowest word error rate (WER) on a large number of language-dataset pairs while remaining competitive with the best-performing systems on high resource while being assessed exclusively on the VAANI benchmark.
Sujith Pulikodan, A. Basu, J. Pavankumar et al.· 1 citation· ⚡1
Artificial Intelligence has significantly boosted the process of Machine Translation due to its rapid advancement. In this paper we present a study on the ability of pre-trained deep learning models for speech translation when applied in intelligent and industrial communication systems. There are significant advances in automatic speech recognition (ASR) and speech-totext systems at this time. The extensive use of data for training such models has been proved to be the key to the most recent advances in speech transcription. However, some of these tools have not achieved the same degree of accuracy when it comes to some Indian languages. While, for example, Whisper or Wav2Vec 2.0 achieve good performance on speech-to-text conversion and multilingual efforts such as NLLB are making strides beyond language boundaries; issues remain, particularly for less commonly spoken indigenous languages. With the advent of speech to speech translation, evaulation of its performance continues to be challenging- due to the reliance upon reference translations, which are scarce for many Indian languages. This is an investigation to see if Whisper performs well with Hindi and Tamil speech and Wav2Vec 2.0. Different segments (in length and flow) from LibriSpeech were used as audio input for both systems. Score is computed by word error rate and BLEU, and human review having closer look at the translation quality. While numbers may help quantify output, meaning is more likely to be evident when a person reads the results. Where Wav2Vec 2.0 fails to sustain, Whisper holds up: particularly for sounds that linger, that is. One falters with changes in pitch or changes in loudness; the other can keep up. Garbage in is garbage out - errors early in the process of converting speech to text can only get worse in the process of converting the latter to speech. The ability to process everything simultaneously saves times, which can be crucial when timing is a critical. Pre-built AI systems have demonstrated their ability to transcend languages, seamlessly integrating into active workspaces where conversations can freely flow.
D. P., Elakkiya A, Aparna Pv· International Conference on...· 0 citations
We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-based ASR models such as Whisper have revolutionized ASR, but most prominent models are highly multilingual. As a result, these models often perform poorly on languages less well-represented in their training set. While it has long been known that effective language adaptation can be achieved through simple fine-tuning on monolingual data, this strategy has only been applied to a small number of languages. We massively scale up this simple approach to 102 languages covered in the FLEURS dataset, while also implementing a more complex language adaptation strategy that integrates monolingual tokenizer replacement and data augmentation using text-only fine-tuning. BuzzASR models outperform Whisper-large-v3 on 77 out of 102 languages, reducing character error rates (CER) by a factor of over 2.8 on average. Our models achieve state-of-the-art CER among open-source systems on 27 of 102 languages on the combined FLEURS and Common Voice test set. Our tokenizer replacement strategy yields an average 3.3x improvement in compression rate (characters per token) over Whisper's multilingual BPE, with gains of up to 21.7x. We release all models, code, and detailed results: https://lemn-lab.github.io/buzz-asr
Shivam Singh, Aditya Yadavalli, Catherine Arnett et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
