Conversational Facial Dynamics as Behavioral Identity Signals: From Naturalistic Conversations to Avatar-Mediated Communication
Abstract
People differ not only in what they express facially during conversation, but also in how their facial behavior unfolds over time. This person-specific temporal structure remains underexplored computationally. Prior work has largely focused on controlled settings with limited numbers of participants, and many facial representations entangle motion with static facial geometry, making it unclear whether recognition performance can be attributed to dynamics rather than appearance. We address these limitations through a shape-excluded analysis of facial communication dynamics. Specifically, we extract per-frame FLAME (Faces Learned with an Articulated Model and Expressions) parameters from video and retain only the expression and jaw parameters (FLAME-EJ) while discarding facial shape; the resulting frame sequences form a dynamics-focused representation of how facial behavior evolves over time. We evaluate this representation in two complementary settings: avatar-mediated facial reenactment in NV-FAIR (161 identities; more than 650,000 synthetic clips) and spontaneous dyadic conversation in CANDOR (1,429 speakers; 87,446 training utterances). In both settings, we use the same two-stage Conformer-based pipeline, with supervised contrastive learning followed by task-specific focal-loss adaptation. On NV-FAIR, the proposed representation supports recovery of the hidden driving identity, reaching a competitive 0.844 AUC with the first-order temporal-difference variant of the expression+jaw trajectory (FLAME-EJ VEL) and 0.773 AUC with FLAME-EJ RAW; the stronger performance of FLAME-EJ VEL is consistent with the avatar setting, where temporal differencing suppresses slowly varying target-specific effects while preserving driver-transferred motion. On CANDOR, the same pipeline achieves 60.09% identification accuracy and 60.34% macro-F1 using FLAME-EJ RAW, and in this setting, FLAME-EJ VEL does not improve over FLAME-EJ RAW. Together, these results show that conversational facial dynamics retain measurable person-specific information even when static facial shape is removed or replaced. This has implications for behavioral identity modeling in multimodal interaction and for identity leakage in avatar-mediated communication.