Recurrent neural network-based collaborative perception method for robot multimodal sensors

Abstract

With the advancement of industrial automation and intelligent manufacturing, robots are increasingly requiredtoperform collaborative perception of their own operating status and external working environment in scenarios suchasassembly, inspection, operation & maintenance and human-robot collaboration. However, multi-modal sensors onrobotsexhibit significant heterogeneity in sampling rate, physical dimension, noise characteristics and information density, which makes it difficult for conventional time-series models to simultaneously cope with long-term dependencies, cross-modal coupling and physical consistency. This paper proposes a recurrent neural network based collaborative perceptionmethod for robot multi-modal sensors. The method builds an internal multi-physics branch and an external context branch to perform intra-modal temporal modeling and semantic infomation extraction respectively; a three-headattention mechanism (self, external and additive) is introduced to achieve dynamic cross-modal fusion; a DiffusionConvolutional Recurrent Neural Network (DCRNN) is further coupled to realize joint spatio-temporal modeling over thesensor node graph; self-supervised pre-training and multi-task learning are adopted, with the residual of partial differential equations introduced as a physical consistency loss. Experiments on five industrial datasets across fourprediction horizons show that the proposed method consistently outperforms Prophet, single-branch LSTMand DCRNNbaselines on MAE, RMSE, MAPE and R² metrics, and the advantage becomes more significant as the prediction horizongrows, verifying the effectiveness of the "recurrent backbone + attention fusion + physical constraint" frameworkforrobot multi-modal collaborative perception.