Vision-Based Human Action Recognition for Automated Maintenance Operation Detection in Industrial Equipment Using Machine Learning Algorithms

Abstract

The maintenance and repair of industrial equipment is a process whose automation can improve the efficiency and safety of industrial production. This paper presents a software service that recognizes service engineer actions from video captured during equipment maintenance. The development stages of this service are described, along with the key features selected for training the machine learning model and their use in implementation. Experiments were conducted to test the software service under laboratory conditions, using the disassembly of a centrifugal oil pump and the replacement of a device in a smart panel for electricity distribution and metering as case studies. The trained model recognizes the required actions with at least 87% accuracy. Finally, the paper describes the testing and hardware requirements, the main functionality, and the advantages of using the service. The results show that identifying the actions of a service engineer requires only the position of their hands in two coordinates, x and y. Dimensionality reduction methods require the extraction of three types of features—geometric, angular, and distance—as well as the use of statistical parameters of time series data in conjunction with the energy and entropy of the wavelet transform coefficients of the time series data. These transformations enable the use of simplified machine learning methods and do not require high-powered devices on-site. It is shown that the presented method allows for a significant reduction in the incoming information for the service to operate (by 31.72 times), which is expressed by storing the coordinates of the hands instead of the video (5.9 MB versus 186 KB).