UniEE: Heterogeneity-Aware Engagement Estimation across Multimodal Feature Representations, Interaction Structures, and Label Semantics

Abstract

Engagement estimation is often formulated as continuous framelevel regression, yet real-world interaction datasets differ not only in domain distribution but also in their multimodal feature representations, social interaction structures, and label semantics. Some datasets annotate continuous engagement trajectories, whereas others provide categorical labels for task or social engagement. We introduce UniEE, a unified framework for heterogeneity-aware engagement estimation that learns a shared engagement representation from heterogeneous multimodal feature streams, dyadic and multi-party interactions, and continuous and categorical supervision. Hierarchical feature-group fusion organizes 11 feature streams into semantic groups and performs intra-group and inter-group selfattention, while attention-based multi-partner pooling generalizes dyadic interaction modeling to multi-party conversations. As an exploratory mechanism for bridging label semantics, a regression–classification bridge maps categorical engagement probabilities to a pseudo-continuous scale. In the MultiMediate'26 challenge, UniEE achieves a mean CCC of 0.737 across the four continuous domains, the highest in the reported public leaderboard snapshot, with particularly strong performance on the multi-party MPIIGroupInteraction domain. On PInSoRo, UniEE improves mean Cohen's ?? from 0.127 to 0.224, representing a 76% relative improvement over the released best-combined baseline. Source code is available at https://github.com/Yuefeng-Zou/UniEE.