Combining exploration and imitation in contact-rich task learning on an articulated soft robot arm

Abstract

Learning from demonstration (LfD) has become a popular approach with the emergence of modern transformer-based algorithms. However, the performance of these policies is limited by the quality of the demonstrations. Combining imitation and exploration promises to train policies that perform better and are more reliable. However, this requires a robotic system that can explore safely without damaging itself or the environment, especially in contact-rich tasks during which the robot must exert force on its environment to solve the task. In this study, we investigate the combination of a state-of-the-art reinforcement learning (RL) algorithm with human demonstrations to learn how to open a door with minimal task-specific engineering on an articulated soft robot arm. We found that learning from both exploration and demonstration data stored in separate buffers makes the algorithm not only more sample-efficient and robust but also allows the policy to reach a higher performance level than the provided expert demonstrations. We also show that using an articulated soft robotic arm allows us to perform RL on a real robotic system without any pretraining and with a simple safety system that does not require any additional sensors, such as force–torque sensors. Additionally, we can implicitly learn the nonlinearities stemming from the soft materials in the actuator. Our findings show that combining LfD with RL results in both better performance and more robust behaviors and indicate that articulated soft robots allow for learning contact-rich tasks safely on a real system.