SQUAD: A Unified Framework for 3D Dyadic Scene Understanding with Applications to Visual Focus of Attention and Object Use

Abstract

Social dyads are rich in nonverbal behavior. Particularly in parent-child interactions involving toys, the analysis of coordinated behavior can be used to assess the child’s development, and the relation between the two interactants. Estimating nonverbal cues in such naturalistic settings using automated, camera-based methods remains challenging. The reliance on a single-camera view reduces the robustness and utility of measurements, as it is prone to occlusions, motion of the interactants and the camera, and insufficient resolution. To address these challenges, we focus on capturing interactions in 3D. We introduce SQUAD: Scene QUantification for the Analysis of Dyads, a 3D scene reconstruction tool focusing on naturalistic dyadic interactions recorded from multiple views. SQUAD estimates the 3D positions of body keypoints and object centers from continuous interactions without assuming calibrated cameras. We demonstrate the merits of 3D analysis using SQUAD with the automated analysis of visual focus of attention and object use, for which we develop simple modules. We evaluate our approach on challenging, free-play parent-child interactions involving 10-month-old infants and age-appropriate toys. Our experiments reveal that 3D analysis of dyads can be robust and that SQUAD can flexibly accommodate multimodal analyses and allows for the extension to a range of other nonverbal, individual or coordinated, behaviors.