Distributed Reinforcement Learning for Cooperative UAV-UGV Formation Control with Multirate Dynamics

Abstract

Coordinating heterogeneous aerial and ground vehicles with fundamentally different dynamic time-scales is a longstanding challenge in multi-agent robotics. This paper proposes a distributed reinforcement learning (DRL) framework for cooperative Unmanned Aerial Vehicle (UAV) and Unmanned Ground Vehicle (UGV) formation control that explicitly accounts for multirate dynamics: UAVs execute high-frequency control at 10 Hz while UGVs operate at 2 Hz. Each agent maintains a local actor-critic network trained via a variant of MultiAgent Proximal Policy Optimisation (MAPPO) augmented with a consensus-based communication graph and a multirate synchronisation module that bridges the temporal mismatch between the two vehicle classes. A shaped reward formulation penalises formation deviation, inter-agent collisions and communication dropout simultaneously. Simulation experiments In a physicsaccurate Gazebo/ROS2 environment with two UAV leaders, two UAV followers, and two UGVs demonstrate a task success rate of 96.4%, a formation accuracy of 97.1%, and a collision rate of only 0.8%, outperforming MADDPG, MAPPO (singlerate), centralised DDPG and PID baselines by margins of up to 24.9 percentage points. The framework is further validated in a disaster-response scenario with a dynamic obstacle fields confirming its suitability for real-world heterogeneous multirobot missions.