An Attention-Based Multimodal Sensor Fusion Framework for Intelligent Human Activity Decision-Making System

Abstract

Human Activity Recognition (HAR) is a crucial subject in the field of intelligent healthcare, smart environments, assisted living systems and context-aware decision-making applications. Radio-enabled smartphones and wearable devices have made it possible to gather multimodal sensor data continuously, offering novel possibilities for accurate activity monitoring and behavioral analysis. This paper introduces an Attention-Based CNN-BiLSTM Multimodal Fusion Framework for intelligent human activity recognition and decision-making. The framework proposed uses accelerometer and gyroscope sensor signals that are extracted from the University of California, Irvine (UCI) Human Activity Recognition dataset which consists of six classes of activities that are performed by the human body in a dynamic and static manner in daily life. First, one-dimensional Convolutional Neural Networks (CNNs) are used to learn discriminative spatial features from each sensor modality. Then, Bidirectional Long Short-Term Memory (BiLSTM) networks are used to represent the temporal relationships between activities in the activity sequence, by incorporating contextual information in both forward and backward directions. An attention mechanism is added to selectively extract and highlight informative motion features and prevent irrelevant sensor information. The enhanced representations of attention are finally fused in a cross-modal fusion layer to create a holistic activity representation for the classification. Experimental results show that the proposed framework reach an accuracy of 97.8%, precision of 97.2%, recall of 97.1%, F1-score of 97.1% and AUC of 0.98. From the comparative and ablation analyses, it can be seen that the attention-guided multimodal fusion method is superior to the traditional machine learning and deep learning methods. The proposed framework has shown that it is reliable, scalable and efficient solution to the intelligent activity-aware decision-making systems in real-life smart applications.