Human-as-a-Sensor: Inferring Individual Engagement from Teammates Alone via Latent Group State Modeling

Abstract

Can an individual’s internal state be inferred in the absence of direct observation? This question is increasingly critical in multimodal interaction settings where privacy constraints, occlusion, or sensor failure render self-centric sensing unreliable or unavailable. In this paper, we introduce a novel task formulation for multimodal interaction analysis, termed human-as-a-sensor, in which the state of a target individual is inferred exclusively from the multimodal behavior of their teammates. We investigate this problem in the context of group collaboration, focusing on the estimation of individual engagement and emergent leadership. To address the inherent challenges of this setting, we propose a Latent Group State Model (LGSM), a structured framework that first encodes teammates’ multimodal signals into a shared latent representation capturing group interaction dynamics, and subsequently performs target-conditioned inference. We evaluate the proposed formulation and model on a multimodal dataset of three-person teams. Our results demonstrate that LGSM significantly outperforms conventional machine learning, deep learning, and vision-language model baselines in the others-only setting. Analyses indicate that performance gains are most pronounced in interaction-dense segments, that multimodal fusion plays a critical role in improvements. Rather than substituting direct sensing, the proposed formulation offers a complementary paradigm for privacy-aware and robust multimodal interaction analysis in scenarios where direct observation is infeasible or intentionally restricted.