Skip to content

EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing

Aug 2026 · arXiv.org · Vol abs/2608.18832 · 1 citation · 41 references
Computer Science

TL;DR

This work contends that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain, and presents EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them.

Abstract

Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.

View source

Similar papers

Sep 2026

ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarios

Lip synchronization aims to generate visual lip dynamics that align precisely with speech audio. Despite the high generation quality of diffusion models, they often struggle in complex scenarios and suffer from prohibitive inference latency, limiting real-world deployment. We present ComplexSync, a unified diffusion-ba...

Jia-Ran Cai, Xing-Pei Ma, Shen-Neng Huang · 0 citations
Preprint Nov 2026

VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing

Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when drivi...

Ye-Xin Liu, Yun-Long Lin, Zehan Wang et al. · 0 citations
Sep 2026

TexTailor: Texture-Preserving Video Virtual Try-On via Adaptive Garment Conditioning

Video virtual try-on has attracted increasing attention due to its broad potential in digital fashion and intelligent e-commerce. However, existing methods primarily focus on low-resolution settings and still face substantial challenges when extended to high-resolution scenarios. These limitations can be attributed to...

Zi-Jing Qin, Jun Zhou, Rui-Cheng Zhang et al. · 0 citations
Preprint Nov 2026

TaoMate-FVSR: Audio-Guided Facial Video Restoration

Low-quality facial videos suffer from missing fine details, while AI-generated videos may additionally contain implausible teeth and lip movements that do not match the speech. Restoring the mouth region is particularly challenging: enhancing existing details can preserve structural errors, whereas regenerating them ri...

Xin-Yang Li, Ze Xu, Jing-Na Sun et al. · 0 citations
Sep 2026

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and t...

Bang Tang · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.