Quality-Diversity Reinforcement Learning using Behavior Regulated Policy Gradient

Abstract

The family of Quality-Diversity algorithms, such as MAP-Elites, aims to generate a large collection of diverse and highperforming solutions using Evolutionary Computation. Despite their success in domains like evolutionary robotics, relying heavily on random mutations inspired by Genetic Algorithms (GA) makes MAP-Elites inefficient in evolving high-dimensional solutions. This limitation motivated the creation of PGA-MAP-Elites, which incorporated policy gradient (PG) from deep reinforcement learning and enabled evolution on large neural networks. DCRL-MAP-Elites, the latest successor of PGA-MAPElites, further advanced the performance leveraging an actor-critic training that is conditioned on the descriptor. However, the critic evaluation in DCRL-MAP-Elites is made in a non-Markovian manner, which could mislead the neuroevolution with incorrect gradient signals. Therefore, we introduce BRPG-MAP-Elites, a new QD-RL algorithm adopting a strictly Markovian actor-critic architecture within the MAP-Elites framework. BRPG-MAP-Elites learns to predict the impact of actions on both behavior and fitness, using this information to create alternative descriptor-conditioned mutations. Following a comprehensive evaluation on a wide range of locomotion control tasks, our method demonstrates a 43% improvement in average QD-scores over DCRL-MAP-Elites and achieves higher robustness in the generated policies.