A multi-agent guided soft actor–critic (MAGSAC) deep reinforcement learning algorithm to enable multiple UAVs to simultaneously arrive at multiple constant-velocity moving targets and outperforms existing mainstream algorithms in synchronization success rate, temporal synchronization accuracy, and safety.
Abstract
Multiple unmanned aerial vehicles (UAVs) performing cooperative missions in complex environments face challenges such as difficult cooperative decision-making, stringent spatiotemporal consistency constraints, and environmental uncertainty. The cooperative mission considered in this paper aims to enable multiple UAVs to simultaneously arrive at multiple constant-velocity moving targets. To address these challenges, this paper proposes a multi-agent guided soft actor–critic (MAGSAC) deep reinforcement learning algorithm. Under the centralized training with decentralized execution (CTDE) framework, a Guider network is introduced to guide the local actor network in learning coordinated strategies, thereby alleviating the non-stationarity of multi-agent decision-making under uncertain environments. An estimated time of arrival (ETA)-based spatiotemporal coordination reward function is designed to promote synchronized arrival. To address sparse rewards, a hindsight experience replay (HER) mechanism based on backward trajectory reconstruction is developed, and a delayed collision-constraint activation mechanism is incorporated to improve convergence while maintaining flight safety. Simulation results show that MAGSAC outperforms existing mainstream algorithms in synchronization success rate, temporal synchronization accuracy, and safety.
An efficient way to resolve the curse of dimensionality, improve obstacle avoidance and cooperative formation control of UAVs was found and shows great prospects of practical application in such domains as military operations, search and rescue missions, transport automation and disaster management.
Qadir Talibov· Problems of Information Tech...· 0 citations
A cooperative guidance law based on the experience-guided multi-agent proximal policy optimization (E-MAPPO) algorithm is proposed for multiple unmanned aerial vehicles (UAVs) to track dynamic points of interest in civilian applications, such as collaborative search and rescue and environmental monitoring. In multi-UAV cooperative tracking, accurate arrival-time coordination is important for improving collaborative task execution, but it remains challenging because of continuous action spaces, target maneuvering, uncertain time-to-go estimation, and inefficient exploration in multi-agent reinforcement learning. Specifically, a multi-UAV cooperative guidance environment is formulated, and the problem is modeled as a Markov decision process. To address the challenges of large action spaces and poor convergence in multi-agent reinforcement learning, an experience-guided MAPPO framework is introduced to enhance training efficiency and policy stability. Different from standard MAPPO, the proposed E-MAPPO introduces proportional-navigation-guided experience only during the early training stage to guide exploration, while the final policy is still optimized through the MAPPO objective. Subsequently, a composite reward function is designed by integrating distance-based heuristic terms with auxiliary guidance signals, thereby improving exploration efficiency and facilitating coordinated rendezvous and tracking of dynamic references. Comparative simulations with cooperative proportional navigation guidance (CPNG), sliding mode control (SMC), and standard MAPPO are conducted under different target motion scenarios. The results show that E-MAPPO reduces the average convergence step by 17.07% compared with MAPPO. In the straight-moving target scenario, E-MAPPO reduces the cooperative time error by 55.10% compared with CPNG and by 8.33% compared with MAPPO. In the S-type maneuvering target scenario, E-MAPPO reduces the cooperative time error by 55.81% compared with CPNG and by 9.52% compared with MAPPO. Monte Carlo experiments further verify its effectiveness and robustness. Additional robustness tests under Gaussian measurement noise, observation bias, and communication delay show that the proposed method maintains acceptable tracking accuracy and cooperative timing performance under different uncertainty conditions. In addition, the results indicate that the proposed method generalizes well to different types of maneuvering targets.
Hao Xiong, Minghu Tan, Xiaoyu Liu et al.· Drones· 0 citations
Reinforcement learning techniques have been widely applied to multi-UAV cooperative path-planning tasks. However, existing multi-agent reinforcement learning methods are still affected by environmental non-stationarity, cooperation difficulties among agents, and low utilization efficiency of experience samples in complex obstacle environments. These issues often lead to slow convergence and unstable training performance. To address these problems, an Improved Experience Replay Multi-Agent Deep Deterministic Policy Gradient (IER-MADDPG) algorithm is proposed for multi-UAV cooperative path planning. First, a cooperative path-planning model is established under the Centralized Training Distributed Execution framework. Second, a dual-layer replay buffer structure consisting of a global replay buffer and a local replay buffer is designed to preserve both global cooperative information and individual experience. Third, a fusion experience sampling mechanism is introduced by combining prioritized experience replay and random uniform sampling to improve sample utilization efficiency and training stability. Finally, training experiments were conducted in environments with different obstacle configurations to evaluate the proposed method. Experimental results demonstrate that IER-MADDPG outperforms other comparison algorithms in terms of convergence speed, training stability, and path-planning performance.
Long Wen, Hui Tan, Yuxi Liu et al.· Italian National Conference...· 0 citations
Deploying Uncrewed Aerial Vehicles (UAVs) for dynamic target search in disaster response scenarios can reduce losses. This paper investigates multi-UAV cooperative trajectory planning for dynamic target search in a three-dimensional environment with static obstacles, aiming to maximize the number of searched targets and minimize the average uncertainty of the search area, while ensuring collision avoidance between UAVs and obstacles. Existing Multi-Agent Reinforcement Learning (MARL) based methods face the sparse reward problem in dynamic target search, which hinders planning feasible multi-UAV trajectories. Notably, Large Language Models (LLMs), with extensive pre-trained knowledge and powerful semantic reasoning capabilities, exhibit potential for designing high-quality reward functions to alleviate the sparse reward problem. Therefore, we propose an LLM-guided Multi-Agent Proximal Policy Optimization (LLM-MAPPO) algorithm, which leverages LLMs’ reasoning capabilities to guide MARL policy learning and plans multi-UAV trajectories for efficient dynamic target search. Specifically, we design an offline LLM reward shaping scheme that generates dense reward signals to mitigate the sparse reward problem. Moreover, we propose a dual-mode pheromone-based search mechanism to guide UAVs to respond promptly to changes in target positions. Experimental results demonstrate that LLM-MAPPO significantly outperforms compared algorithms in terms of the number of searched targets and average area uncertainty, while successfully avoiding collisions. In particular, LLM-MAPPO reduces the target search time by 71.4%.
Yifei Liu, Xiaoshuai Li, Xia Jiang et al.· IEEE Transactions on Cogniti...· 0 citations
Coordinating heterogeneous aerial and ground vehicles with fundamentally different dynamic time-scales is a longstanding challenge in multi-agent robotics. This paper proposes a distributed reinforcement learning (DRL) framework for cooperative Unmanned Aerial Vehicle (UAV) and Unmanned Ground Vehicle (UGV) formation control that explicitly accounts for multirate dynamics: UAVs execute high-frequency control at 10 Hz while UGVs operate at 2 Hz. Each agent maintains a local actor-critic network trained via a variant of MultiAgent Proximal Policy Optimisation (MAPPO) augmented with a consensus-based communication graph and a multirate synchronisation module that bridges the temporal mismatch between the two vehicle classes. A shaped reward formulation penalises formation deviation, inter-agent collisions and communication dropout simultaneously. Simulation experiments In a physicsaccurate Gazebo/ROS2 environment with two UAV leaders, two UAV followers, and two UGVs demonstrate a task success rate of 96.4%, a formation accuracy of 97.1%, and a collision rate of only 0.8%, outperforming MADDPG, MAPPO (singlerate), centralised DDPG and PID baselines by margins of up to 24.9 percentage points. The framework is further validated in a disaster-response scenario with a dynamic obstacle fields confirming its suitability for real-world heterogeneous multirobot missions.
Sankara Rao, P. Palpandi, G. Beulah et al.· International Conference on...· 0 citations
Cooperative navigation of multi-agent UAVs in complex environments faces key challenges including local optima traps, sparse rewards, learning imbalance among agents, and insufficient cross-scenario generalisation. This paper proposes a multi-agent deep reinforcement learning framework that addresses these issues through coordinated exploration, demonstration exploitation, safe curriculum scheduling, and structure-aware generalisation. First, a perception mechanism combining memory of visited states, directional novelty estimates, and penalty backpropagation enables agents to proactively detect and escape local optima. Second, a hierarchical collaborative demonstration buffer with tiered behaviour cloning manages trajectories by degree of team collaboration and applies differential supervision to the actor network, improving demonstration utilisation under sparse collaborative signals. Third, a safety-aware dual-condition curriculum scheduling mechanism reviews mastered scenarios through back-testing and experience pre-filling during training, suppressing catastrophic forgetting while ensuring both task performance and flight safety. For generalisation, local geometric features computed from sensor readings are abstracted into a domain parameter, through which a structure-aware gating network and mixture-of-experts mechanism condition the policy on local structural patterns rather than scenario-specific coordinates, enabling cross-scenario transfer without exposure to the target environment. The framework is further validated under mixed static-dynamic obstacle settings, showing robust adaptability to dynamic disturbances. Simulation results confirm strong performance in collaboration success rate, navigation robustness, zero-shot cross-scenario generalisation, and dynamic environment adaptability.