A Data-Centric LoRA Recipe for Multilingual Conversational Speech Recognition

Abstract

In this paper, we describe Cake By VPBank’s system submitted to Task I - Multilingual Conversational Speech Recognition of the 2nd MLC-SLM Challenge, reaching 15.84% time-constrained permutation word error rate (tcpWER) on the blind 21-variant, 14-language evaluation set. Rather than change the model architecture, we adopt a strictly data-centric recipe on top of a single LLM-based speech model (VibeVoice-ASR) adapted with Low-Rank Adaptation (LoRA). Our contributions are four-fold: (i) a staged merge-then-fresh LoRA schedule that folds a converged adapter into the frozen backbone and trains a fresh adapter on top, refreshing capacity without catastrophic forgetting; (ii) reference-completeness recovery , which repairs systematically missing speech labels and mutes audio the references cannot account for; (iii) silence-robustness augmentation that makes the model robust to the inter-turn digital-zero regions typical of silence-gated conversational evaluation audio; and (iv) a robust long-form inference and post-processing pipeline. The single model jointly transcribes and attributes both speakers with timestamps, requiring no external diarizer.