Abnormal behavior perception in elderly care scenarios based on multimodal generative adversarial networks

Abstract

In view of the limitations of the existing elderly care anomaly perception methods, which mostly rely on single visual mode and shallow stitching multi-source sensing features, such as insufficient accuracy of long-time progressive anomaly recognition, easy mode collapse of multimodal training, and insufficient utilization of non visual sensing information, this paper proposes a multimodal wgan GP anomaly perception method combining CNN-LSTM spatio-temporal double branch coding. The framework integrates five kinds of heterogeneous sensor data, including vision, inertial IMU, bed pressure, physiological signs and environmental audio, and adaptively allocates the weight of each mode to complete the cross modal feature alignment based on the dual attention of space and time; The shared generator based on the fusion of spatial and temporal two-way sub generators only learns the distribution of normal behavior, and uses multimodal attention discriminator to distinguish the true and false features; Wgan GP gradient penalty and modal imbalance correction item were introduced to stabilize the confrontation training, and the weighted reconstruction error and discriminant confidence were fused to construct a joint abnormal score. Four core abnormalities, including falling, falling out of bed, long-term wandering and physiological index disorder, were identified by dynamic threshold. The experimental results show that the AUC roc of the complete model in this paper can reach 0.978 ± 0.003, the F1 score is 95.64 ± 0.47%, the false positive rate is 2.13 ± 0.24%, and the false negative rate is 3.05 ± 0.30%; Ablation experiments showed that F1 score decreased significantly after removing inertia, bed pressure or audio modes, which confirmed that such non visual modes had complementary contributions to information, and the complete model had stronger scene adaptability and engineering landing value. Not applicable.