Waypoint Navigation of a 2D Drone in Stochastic Environment via Regularised Proximal Policy Optimisation
TL;DR
A three‐stage training pipeline that combines imitation learning and reinforcement learning for 2D drone waypoint navigation is presented, finding that training reward and critic explained variance are unreliable proxies for navigation quality, underscoring the need to evaluate policies on held‐out targets.
Abstract
Autonomous navigation of aerial vehicles under stochastic wind disturbances is a challenging control problem that classical fixed‐gain methods struggle to handle reliably. We present a three‐stage training pipeline that combines imitation learning and reinforcement learning for 2D drone waypoint navigation: a cascaded PD expert collects quality‐filtered demonstration trajectories; behaviour cloning (BC) warm‐starts the policy by supervised imitation of those demonstrations; and Proximal Policy Optimization (PPO) fine‐tunes it through environment interaction, with an auxiliary BC loss retained at every update to prevent the RL gradient from erasing the imitation prior. Trained with randomly sampled targets, the policy generalises to unseen waypoints under continuously drifting wind, maintaining stable, near‐level flight. A systematic ablation across three random seeds shows that the strength of the auxiliary BC loss is the most critical design choice: removing it yields a policy that maximises shaped reward yet fails to navigate, crashing on nearly every episode, whereas over‐weighting it caps improvement near the demonstrator's level. Network capacity has negligible impact, whereas larger rollout buffers consistently yield the best and most consistent navigation. We further find that training reward and critic explained variance are unreliable proxies for navigation quality, underscoring the need to evaluate policies on held‐out targets.