LoRA-MoE Fine-Tuning for Improved Speech Recognition in People With Parkinson’s Disease
TL;DR
LoRA-MoE, a parameter-efficient adaptation method that combines Low-Rank Adaptation with a mixture of experts (MoE) to improve speech recognition for individuals with Parkinson’s disease (PD), demonstrates consistent improvements and stable performance across all severity levels, and its performance is robust to the number of experts.
Abstract
Despite advances in automatic speech recognition (ASR) technology, its performance remains limited when processing dysarthric speech, which hinders people with dysarthria from accessing voice-based digital technologies. In this study, we propose LoRA-MoE, a parameter-efficient adaptation method that combines Low-Rank Adaptation (LoRA) with a mixture of experts (MoE), to improve speech recognition for individuals with Parkinson’s disease (PD), who often exhibit hypokinetic dysarthria. We integrate LoRA-MoE into the Whisper model by freezing its pre-trained weights while introducing multiple LoRA experts and a router network that dynamically combines them based on the input. Experiments on a Korean PD speech dataset show that LoRA-MoE achieves a character error rate (CER) of 12.01%, a 38.54% relative improvement over the baseline Whisper model, and outperforms full fine-tuning and single LoRA adaptation by 46.00% and 11.37%, respectively, with consistent improvements across all PD severity levels. The proposed approach demonstrates consistent improvements and stable performance across all severity levels, and its performance is robust to the number of experts; we adopt three experts as our default configuration.