Interactive Teaching of Robotics Based on Multimodal Human Pose Estimation

Abstract

Traditional robotics courses often provide limited opportunities for engineering practice, making it difficult for students to apply theoretical knowledge to integrated engineering tasks. To address this issue, this study proposes a practical robotics teaching approach integrating multimodal vision and human pose estimation. An RGB-IR multimodal human pose dataset is constructed using an Intel RealSense D435 camera, and an RGB-IR dual-branch pose estimation network with a Cross-Modal Adaptive Gated Fusion (CM-AGF) mechanism is developed based on YOLOv8-Pose. The detected two-dimensional keypoints are combined with depth information for three-dimensional reconstruction and joint-angle calculation, and the resulting human motion parameters are mapped to robot control commands. Experimental results demonstrate the feasibility of integrating multimodal perception, pose estimation, three-dimensional motion reconstruction, and robot control into a unified practical teaching platform. The proposed approach provides students with opportunities to integrate computer vision, artificial intelligence, and robotics in engineering practice.