Cross-Modal Knowledge Distillation From Synthetic Video for IMU-Based HAR
Abstract
Human Activity Recognition (HAR) on resource-constrained edge devices relies on lightweight models, but robustly training these architectures is severely bottlenecked by the scarcity of diverse, annotated Inertial Measurement Unit (IMU) data. To overcome this empirical limitation, we propose a cross-modal Knowledge Distillation (KD) framework that uses the zero-shot generative capabilities of foundational text-to-video diffusion models. By synthesizing complex human kinematics and filtering the outputs for visual coherence and pixel-level temporal stability, we extract spatio-temporal probability distributions to construct class-level Knowledge Banks (KB), both per generator and aggregated into a generalized Ensemble KB. These synthetic visual soft labels, which act as class-level supervisory priors rather than instance-level targets, are then cross-modally distilled into five efficient student architectures, a Bidirectional Long Short-Term Memory (BiLSTM) network, 1D and 2D Convolutional Neural Networks (CNNs), a lightweight hybrid CNN, and a selective state-space model (SSM), using a balanced trust ratio, and evaluated on a subject-independent held-out test partition. Empirical evaluations indicate that the benefit of cross-modal supervision is concentrated in the moderately scarce regime: at 25% data availability the strongest individual teacher (LTX-Video 2.3) raises the BiLSTM Macro F1-Score from 46.66% to 50.78% and 53.88% using fixed and validation-selected trust ratios, respectively (absolute gains of + 4.12 and + 7.22 points), while also recovering the Stairs class from 3.0% to 13.5% in the fixed-ratio case. The effect is neutral once empirical sensor data becomes abundant. These results suggest that simulated visual physics can anchor and regularize empirical sensor networks in the scarce-label regime, bridging the spatial-temporal modality gap without incurring any additional computational overhead at inference.