HRL-Det: Hierarchical Reinforcement Learning for Sequential Object Detection in Aerial Imagery

Abstract

Object detection in unmanned aerial vehicle (UAV) imagery suffers from severe scale variation, dense object packing, and prohibitive computational cost when conventional detectors exhaustively evaluate high-resolution frames. Reinforcement learning (RL)-based sequential detectors offer a promising alternative by formulating localization as an active search process, yet existing methods are limited by discrete-time state transitions, sparse reward signals, and premature policy collapse. In this paper, we propose HRL-Det, a hierarchical reinforcement learning framework that addresses these challenges through two tightly coupled innovations. First, a Neural ODE-driven Continuous-Time Bellman State Evolution module models the agent’s state dynamics as a stochastic differential equation governed by the Hamilton–Jacobi–Bellman equation, enabling fine-grained temporal reasoning with memory-efficient adjoint-based backpropagation. Second, a Lyapunov-Guided Entropy-Regularized Reward Shaping mechanism constructs convergence-promoting dense rewards informed by Lyapunov stability analysis while maintaining exploration diversity through maximum entropy optimization. Extensive experiments on VisDrone2019, DroneVehicle, and MS COCO 2017 show that HRL-Det achieves mAP@0.5 of 0.412, 0.812, and 0.735, respectively, outperforming existing RL-based detectors and achieving competitive accuracy relative to representative non-RL detectors under the same COCO metric, while requiring only 17.3 M parameters and an average of 6.3 search steps per object.