Environmental Context-Aware Human Action Recognition from 4D Millimeter-Wave Radar Point Clouds

Abstract

As an emerging sensing modality, 4D millimeter-wave radar provides a privacy-preserving and lighting-robust solution for indoor scene perception and human action recognition (HAR). However, existing radar-based HAR methods mainly focus on short-term motion dynamics while overlooking the influence of indoor scene context. Furthermore, reconstructing indoor layouts from sparse and irregular radar point clouds remains challenging. To address these issues, this paper proposes an environmental context-aware radar action recognition framework (ECRAR) that jointly models human motion and indoor scene context from 4D millimeter-wave radar point clouds. Specifically, a trajectory-to-floorplan inversion branch reconstructs indoor layouts from long-term trajectory point clouds by exploiting the relationship between human movement distributions and spatial accessibility. A motion feature extraction branch captures discriminative short-term motion representations through multi-scale spatial modeling, while an environmental context fusion module adaptively integrates scene and motion features using cross-attention. We also construct InActivity-Scene, a new dataset with fine-grained scene annotations and nine categories of daily and hazardous human actions collected in multiple indoor environments. Experimental results demonstrate that ECRAR consistently outperforms existing methods, achieving 92.2% recognition accuracy with significant improvements on environment-dependent actions, while ablation studies further verify the effectiveness of each proposed component.