Engagement is Tied to Context: A Configurable Multimodal Architecture for Engagement Estimation Across Adult and Child Interactions
TL;DR
A configurable multimodal architecture comprising four main modules: multimodal fusion, temporal encoding, interaction modeling, and prediction heads is proposed, taking frame-level engagement estimation as the objective and evaluating it across datasets spanning adult dyadic conversation and child interaction in free-play settings.
Abstract
Advances in multimodal predictive systems are ushering in a new era in human interaction with technology. One particular area receiving increasing attention is engagement estimation, a key building block in achieving safe and responsive systems. Engagement, however, is a thoroughly context-dependent phenomenon, varying considerably across cultures, age groups, and specific contexts. In the present study, which is part of the MultiMediate ’26 Grand Challenge, we address this transferability issue in the literature, taking frame-level engagement estimation as the objective and evaluating it across datasets spanning adult dyadic conversation (NoXi+J, MPIIGI) and child interaction in free-play settings (PInSoRo). We propose a configurable multimodal architecture comprising four main modules: multimodal fusion, temporal encoding, interaction modeling, and prediction heads. We report a global score of 0.59 across the evaluations (3rd on the leaderboard for ’26), including a new best score of 0.66 CCC for NoXi-J. The design allows for specific analysis of which predictive information matters across domains, what might be modeled similarly, and what not. To advance both the technical frontier and the psychological understanding of engagement, we further examine annotation uncertainty in PInSoRo, showing that performance drops sharply once annotator-disagreement frames are included. Code is available.