LipAdapter: Text-to-Video Alignment is All You Need for Lip-to-Speech
Abstract
Lip-to-Speech synthesis aims to generate intelligible, natural-sounding, and temporally synchronized speech from silent visual recordings of a speaker. Although recent end-to-end models have shown promising results, they require large amounts of training data and often lack out-of-dataset generalization. In this work, we introduce LipAdapter, a generic and efficient modular framework that adapts an existing frozen pre-trained Text-to-Speech model into a lip-synchronized speech generator. LipAdapter bridges the modularity gap between frozen Visual Speech Recognition and Text-to-Speech models by aligning phoneme representation and lip embeddings via a novel text-to-video alignment module. Our model achieves state-of-the-art performance across multiple benchmarks in English while using 15× less training data than previous methods. We also demonstrate the model’s zero-shot capabilities in multilingual lip-to-speech synthesis in French, German, Spanish and Portuguese. Code and demo are available at: https://lipadapter. github.io/ .