Adaptive Exploration and Curriculum Learning for Multi-Agent Reinforcement Learning

Abstract

Multi-agent Reinforcement learning has gained significant attention for solving decision-making problems involving multiple autonomous agents. However, effective learning in MARL is still difficult due to environments, dependencies between agents, and poor exploration strategies. Although adaptive exploration and curriculum learning methods, such as Reward Prediction Error Adaptive Learning (RPEAL) along with Reward-Shaped Adaptive Curriculum Learning (RSACL), have produced good outcomes in single-agent reinforcement learning, their use in multi-agent contexts has not been thoroughly investigated. In this research, RPEAL and RSACL are introduced. This paper extends the previous single-agent work to the broader realm of cooperative multi-agent reinforcement learning. The introduced adaptive control mechanism are integrated into several popular multi-agent algorithms such as IPPO, CPPO, MADDPG, and MASAC are empirically compared in a standard petting-zoo environments. The experimental evaluation shows gains in these algorithms upon the introduction of adaptive control mechanisms, where the centralized critic outperforms the individual learners in terms of stability and convergence. Unlike previous works, which only considered single-agent reinforcement learning, in this paper we extend the RPEAL and RSACL to the multi-agent domain. To be specific, we propose team reward prediction error modeling with a centralized critic, as well as performance-driven curriculum learning for multi-agents.