DynaID-VAE for Speech-Driven Virtual Anchor Generation: Identity-Disentangled Temporal Memory Variational Modeling

Abstract

Generating a virtual anchor from speech has to satisfy three demands at once: the face must stay recognizable as the same person, the motion has to be temporally coherent, and the lips must follow the audio. Existing methods often fall short on the first two points because identity and expression share one entangled representation, and temporal dynamics are modeled only implicitly. We propose DynaID-VAE to address these problems. At its core is an identity–expression disentangled conditional VAE (DC-VAE) that splits the latent space into a time-varying expression subspace and a static identity subspace, held apart by mutual-information minimization and orthogonality regularization. A temporal memory module (TMM) then regularizes the expression trajectory: a GRU propagates sequential state, attention retrieves from a learnable key–value prototype memory, and residual fusion combines the two. Multiscale adversarial supervision and lip–audio synchronization losses complete the training objective. We evaluate on VirtualAnchor-100, a benchmark we recorded ourselves (100 h, 10 anchors), under two complementary protocols. Cross-identity driving is scored only with non-paired measures, namely lip synchronization, distributional video quality, and identity preservation; full-reference image metrics are confined to a self-reenactment protocol, where a genuine paired ground truth exists. DynaID-VAE outperforms the one-reference baselines Wav2Lip, PC-AVS, SadTalker, and DiffTalk under both protocols and on unseen VoxCeleb2 identities. The margins are stable across five identity-disjoint, nested cross-validation folds and are confirmed by an external SyncNet evaluator that never takes part in training, while the model runs at 41.2 FPS with 14.3 M parameters. Ablations separate the contribution of each regularizer and each TMM component. Linear and capacity-matched non-linear probes quantify the factorization as a large reduction of decodable reference identity; full independence is not claimed. A user study confirms the perceptual gains.