Egocentric Online Action Segmentation via Parametric Context Memory Learning

Abstract

To facilitate smart wearable devices or human-like robotics with real-time first-person perspective perception ability, recent researchers proposed the Egocentric Online Action Segmentation (EOAS) task. It requires models to recognize what is happening in egocentric streaming videos and discriminate the starting and ending times of an activity in a real-time manner. However, compared with offline-recorded exocentric videos, egocentric streaming videos cannot provide equivalent sufficient temporal-spatial cues due to the limited perspective and unknown coming frames. Hence, it raises a high demand for the long-term episodic memory ability of models. To this end, most previous approaches work on compressing long-term memory into feature representations. In this paper, we propose a novel EOAS paradigm, termed Parametric Context Memory Learning (PCML), which integrates episodic memory into learnable parameters and keeps dynamic updates according to real-time frames. Concretely, we design the Parametric Context Perception layer and construct a novel Episodic Semantic Memorization Network (ESMN) based on it, which integrates episodic memory into learnable parameters and keeps dynamic updates with real-time frames. We evaluate our proposed method on three public egocentric streaming video benchmarks including EgoPER, EgoProceL, and GTEA. Extensive experiments demonstrate the ESMN model significantly outperforms recent state-of-the-art methods. Our code is available at https://github.com/XunCHN/PCML