Anticipating Object Interactions Via Aggregation and Distillation of Spatio-Temporal Knowledge From Vision Language Models.

Abstract

The increasing prevalence of wearable cameras has driven the development of egocentric (first-person) systems that assist human activities proactively by anticipating imminent interactions. A central challenge in this domain is active object interaction anticipation from first-person video-predicting what interaction will occur, when it will happen, and where it will take place. This involves forecasting (1) what interaction category (verb-noun pair), (2) when (time-to-interaction), and (3) where (the active object's location, bounding box in the last observed frame). However, existing approaches rely on limited prior knowledge about active objects and their state changes, and they struggle to (1) predict diverse state-change interactions, (2) handle temporal uncertainty of changes, and (3) localize the active object accurately in the presence of spatial distractors. To address these problems, we propose ST-KAD. It consists of a Spatial-Temporal Knowledge Aggregator that integrates rich commonsense priors to enhance what-when-where interaction anticipation and guides the model's attention toward informative cues, and a Teacher-Student Distillation framework that enables efficient inference without access to oracle inputs by transferring knowledge from an oracle-informed teacher model to a query-based student decoder. On two egocentric anticipation benchmarks (Ego4D-STA, EPIC-Kitchens-STA), ST-KAD sets a new state of the art, demonstrating accurate what-when-where prediction of future interactions. Moreover, results on four active object detection benchmarks (Ego4D-AOD, EPIC-Kitchens-AOD, MECCANO, 100DOH) further confirm that our prior-informed aggregation and teacher-student distillation generalize beyond anticipation to spatial localization, validating the generality of the design.