Advances in Multi-Agent Deep Reinforcement Learning: Methods with Applications and Challenges

Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning

Cooperative navigation of multi-agent UAVs in complex environments faces key challenges including local optima traps, sparse rewards, learning imbalance among agents, and insufficient cross-scenario generalisation. This paper proposes a multi-agent deep reinforcement learning framework that addresses these issues through coordinated exploration, demonstration exploitation, safe curriculum scheduling, and structure-aware generalisation. First, a perception mechanism combining memory of visited states, directional novelty estimates, and penalty backpropagation enables agents to proactively detect and escape local optima. Second, a hierarchical collaborative demonstration buffer with tiered behaviour cloning manages trajectories by degree of team collaboration and applies differential supervision to the actor network, improving demonstration utilisation under sparse collaborative signals. Third, a safety-aware dual-condition curriculum scheduling mechanism reviews mastered scenarios through back-testing and experience pre-filling during training, suppressing catastrophic forgetting while ensuring both task performance and flight safety. For generalisation, local geometric features computed from sensor readings are abstracted into a domain parameter, through which a structure-aware gating network and mixture-of-experts mechanism condition the policy on local structural patterns rather than scenario-specific coordinates, enabling cross-scenario transfer without exposure to the target environment. The framework is further validated under mixed static-dynamic obstacle settings, showing robust adaptability to dynamic disturbances. Simulation results confirm strong performance in collaboration success rate, navigation robustness, zero-shot cross-scenario generalisation, and dynamic environment adaptability.

Scalable and Sample-Efficient Multi-Agent Imitation Learning

SPD-MAPPO: Reinforcement Learning With Stochastic Policy Distillation for Multi-Vehicle Coordination in Open-pit Mine

Coordination of multiple autonomous trucks is crucial for enhancing the efficiency and safety of modern mining, yet it is challenged by dynamic vehicle-to-vehicle interactions and the complexity of mining transportation. Conventional rule-based methods struggle to balance efficiency with success rates and lack flexibility in diverse scenarios. While multi-agent reinforcement learning (MARL) shows great promise for cooperative tasks, its application in real world is often hampered by challenges in convergence. To address these challenges, we propose SPD-MAPPO, a novel multi-stage learning framework that transfers expertise from imitation learning(IL) to cooperative ability in MARL. The framework first employs IL to pre-train a policy with basic single-agent driving ability, which is subsequently refined for cooperative behaviors through MARL. Specifically, we design a Stochastic Policy Distillation (SPD) mechanism to bridge the gap between single-agent expertise and multi-agent coordination, and a multi-head critic network to achieve more precise credit assignment. We validate our method in a high-fidelity simulator with a real-world map of mine and a truck dynamics model. Our method outperforms typical rule-based and MARL methods in success rate, efficiency, and operational accuracy.

Collaborative Robot Coordination Using Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) has emerged as an important approach for coordinating collaborative robots in industrial automation, warehouse logistics, healthcare, autonomous vehicles, and distributed robotic systems. Traditional centralized robot coordination methods faced limitations such as poor scalability, low adaptability, synchronization issues, and weak fault tolerance in dynamic environments. MARL overcomes these challenges through decentralized learning, where multiple robotic agents interact with the environment, learn from rewards, and improve coordination strategies autonomously. Before 2019, MARL gained significant attention in applications like cooperative navigation, formation control, multi-robot exploration, task allocation, path planning, collision avoidance, and resource sharing. This survey reviews key MARL techniques including Q-learning, Deep Q-Networks (DQN), policy-gradient methods, actor-critic models, and cooperative game-theoretic approaches for robotic coordination. The study explains a structured MARL coordination framework involving environment modeling, state representation, reward optimization, agent communication, and distributed decision-making. Experimental results show that MARL-based robotic systems improve task efficiency, coordination accuracy, energy optimization, adaptability, and collision reduction compared to centralized or heuristic methods. However, challenges such as communication delays, scalability, reward sparsity, non-stationary environments, and convergence instability still remain. The paper concludes that MARL is a promising solution for future intelligent collaborative robotics and highlights future research directions including federated reinforcement learning, explainable AI, edge-based robotic intelligence, and adaptive swarm robotics for Industry 4.0 applications.

XAI-Driven Multi-Agent Reinforcement Learning for Swarm USV Continuous Multi-Target Hunting

Swarm unmanned surface vehicle (USV) has become a promising solution for maritime security and defense activities. However, the complexity of continuous multi-target hunting presents significant challenges for multi-agent coordination in water environments. This study proposes an XAI-driven multi-agent reinforcement learning (MARL) framework for swarm USVs to perform continuous multi-target hunting. The framework integrates multi-agent deep reinforcement learning (basic MAPPO and MAPPO-LSTM) for cooperative hunting policies with explainable AI (XAI) mechanisms to provide interpretable insights into swarm behaviors and decision strategies. Experimental results in a 3D simulation platform environment demonstrate that MAPPO-LSTM achieves superior interception performance, reducing mean time-to-capture and improving trajectory smoothness compared to the baseline MAPPO. Furthermore, the proposed XAI pipeline combines distance-based importance attribution, influence graph analysis, strategic clustering, and temporal dynamics evaluation to explain agent contributions, inter-agent dependencies, emergent strategies, and efficiency patterns. By providing multi-level interpretability, the framework enhances transparency, trust, and deployment readiness of swarm USVs in dynamic maritime defense scenarios.

Exploring Agent Behavior and Performance in Shortened Simulations via Multiagent Reinforcement Learning

This study explores the potential of Multiagent Reinforcement Learning (MARL) for autonomous navigation in a discrete two-dimensional environment. We design and implement an agent that learns an optimal path through a 5×5 grid via repeated interactions and a reward-based mechanism. Over 500 training episodes, we examine the convergence speed of the learned policy, the stability of agent behavior, and the success rate in reaching the goal. Our approach combines artificial neural networks with a multiagent framework, enabling decentralized decision making and scalable adaptation. We discuss critical factors affecting learning stability, including reward function design and network architecture, and outline avenues for extending the methodology to more complex, real-time tasks. The findings demonstrate MARL’s promise in solving navigation problems efficiently and provide concrete recommendations for tuning training parameters and network structures to enhance performance and robustness.