CoNav-UAV is proposed, which explicitly models the target-oriented vision-and-language navigation task as a Stackelberg game between a high-altitude leader and a low-altitude follower, with the system operating on onboard visual and linguistic inputs alone.
Abstract
Target-oriented vision-and-language navigation (VLN) on aerial platforms is attracting growing attention for missions such as disaster rescue, infrastructure inspection, and security patrol. In this task, an unmanned aerial vehicle (UAV) needs to locate targets given only a concise description of their appearance and surroundings. This requires global exploration and grounding as well as collision-free close-range approach, two interleaved processes difficult to reconcile within a single agent. Most existing methods transfer the ground VLN paradigm to a low-altitude UAV and compensate for its inefficient exploration with external assistance. A recent attempt deploys two UAVs at complementary altitudes yet still relies on privileged information and trains its two agents independently, precluding any mutual adaptation essential for cooperation. Here we propose CoNav-UAV, which explicitly models the task as a Stackelberg game between a high-altitude leader and a low-altitude follower, with the system operating on onboard visual and linguistic inputs alone. To solve this game, we introduce Iterative Stackelberg Learning. The leader's high-level vision-language reasoning is refined via memory-based in-context learning, while the follower's precise motion control is updated via DAgger-style expert distillation. The alternation drives both agents toward a Stackelberg equilibrium. CoNav-UAV consistently outperforms single- and dual-agent baselines across three high-fidelity urban scenes from the AerialVLN benchmark. Success rate improves by up to 30.8 points on the learning scene, and 9.0 points under cross-scene transfer while using about 3x less adaptation data. Further analyses validate the complementary gains of the leader and follower updates and reveal robust gains yet distinct learning dynamics across VLM backbones.
Quadrotor UAVs are increasingly deployed in complex missions that demand reliable autonomous navigation and robust obstacle avoidance. Traditional modular pipelines suffer from cumulative latency, motivating a shift toward end-to-end learning-based methods. However, this paradigm still faces two fundamental challenges. First, Euclidean distance-based guidance is prone to local minima under large obstacles and lacks global planning capability. Second, safety mechanisms built on collision penalties alone cannot anticipate hazards, making them fragile at high speed. To address these issues, we propose an end-to-end differentiable physics framework that combines time-of-arrival fields and control barrier functions. During training, we generate a time-of-arrival (TOA) distance field online on the GPU via the jump flooding algorithm (JFA) as privileged information. A position-level geodesic objective is embedded into the differentiable pipeline through a straight-through gradient. A velocity-aligned yaw strategy keeps the depth camera pointed along the flight direction, mitigating blind spots caused by lateral motion. We further formulate the first-order control barrier function (CBF) as an analytic velocity constraint, driving the policy to learn anticipatory deceleration. At deployment, no privileged supervision is required. The policy operates using only onboard depth and inertial states, with no online map or solver. Extensive experiments show that our method outperforms existing baselines across all tested speeds, with zero-shot sim-to-real transfer reaching 7.8 m/s in outdoor forests.
Yan-Jie Liu, Teng-Da Yang, Zi-Han Li et al.· IEEE Robotics and Automation...· 0 citations
Fine-grained inspection by Unmanned Aerial Vehicles (UAVs) in substation environments faces stringent challenges for autonomous navigation and real-time obstacle avoidance due to dense equipment and narrow spaces. Existing pure local planners are highly prone to getting trapped in dead ends due to limited fields of view, whereas global planners incur high computational overhead and struggle to react to dynamic disturbances during execution. To address this issue, this paper proposes a UAV navigation framework based on privileged learning. During the training phase, a pre-built global point cloud map is utilized as a prior environment, and a traditional heuristic search algorithm is employed to generate an absolutely safe, collision-free reference trajectory as an expert demonstration. During the deployment phase, the student policy relies solely on spherical depth and intensity images projected from a Mid-360 LiDAR, along with real-time attitude information provided by an Inertial Measurement Unit (IMU). A two-branch lightweight network is utilized to extract local spatial geometric and dynamic features to predict the trajectory. Simulation experiments demonstrate that this method significantly improves obstacle avoidance foresight, navigation success rate, and trajectory smoothness in complex substation scenarios while maintaining extremely low onboard computational overhead.
With the rapid advancement of autonomous flight technology, there is an increasing demand for higher precision and advanced navigation techniques, with permissible distance errors often restricted to a few meters. Furthermore, the ubiquitous deployment of Unmanned Aerial Vehicles (UAVs) necessitates the integration of novel technologies to ensure operational continuity under adverse conditions, such as environmental signal interference, hostile attacks, or traversal through zones of complete signal loss. This study presents a methodology to address these challenges, enabling themaintenance of coordinates and navigation for UAVs to traverse jammed or completely out-of-coverage zones, thereby avoiding the need for emergency landings or Return-to-Home (RTH) protocols common in current UAV systems. The proposed approach leverages the TransGAN model, a framework typically employed for data analysis comprising a Generator and a Discriminator. In this context, the model processes sequential real-world coordinate data. Under normal GNSS operation, TransGAN is trained as a high-precision prediction model utilizing velocity and coordinate data as inputs. Conversely, during GNSS outages or interference, the trained TransGAN model is utilized to generate coordinates, thereby maintaining navigation capabilities for the UAV.
Nga Vu Quynh, P. N. Huu, Thanh Han-Trong· IEEE International Conferenc...· 0 citations
Air-Ground Object Search (AGOS) in urban environments is a challenging embodied task, which requires an Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV) to jointly search for and verify a specified target vehicle from multi-view visual references. To study this underexplored problem, we introduce AGOS-Bench, the first dedicated benchmark for evaluating whether general-purpose Vision-Language Models (VLMs) can integrate aerial discoveries and ground-level verification through UAV-UGV cooperation. We further provide AGOS-Dataset as the companion resource of exemplary trajectories constructed by an automatic pipeline. It consists of 7.7k episodes for searching objects of diverse categories and attributes, spanning three difficulty levels. To address the AGOS task, we propose AGOS-Agent, a training-free and tool-augmented approach. The agentic method relieves VLMs from complex and dynamic coordination via a deliberate search-handoff-verify cooperation protocol, only demanding VLMs for scene understanding and decision-making. Extensive experiments on nine VLMs show that AGOS-Agent improves overall success rate for eight of the nine evaluated backbones while reducing decision steps for all nine. On the hard split, the SR and SPL of Gemini-3.6-Flash increase from 8.6% to 55.7% and from 7.6% to 44.0%, respectively.
FALCON-MASAC is presented, a safety-integrated multi-agent reinforcement learning framework that decomposes this task into four complementary layers: a hierarchical leader-follower paradigm that pairs a pre-trained virtual leader with followers learning a distributed cooperative policy, and a bypass-side commitment coordination layer that suppresses trajectory chattering and mitigates crossing conflicts among neighboring UAVs.
Yiming Shang, Chang-Ping Du, Rui Yang et al.· Unmanned Systems· 0 citations
We present a learning-based approach to kinodynamic online motion planning for an Unmanned Aerial Vehicle (UAV) operating at a fixed altitude in unknown dynamic environments, where real-time avoidance of both static and dynamic obstacles must be achieved under conditions of extreme partial observability. The UAV is controlled with a single degree of freedom (yaw only), resulting in constrained, nonholonomic motion similar to fixed-wing platforms. The proposed framework integrates a behavior grid map representation with Deep Reinforcement Learning (DRL), using Proximal Policy Optimization (PPO) for stable policy learning in continuous control. The key idea is the co-design of a state representation and control policy that enables reliable navigation using only three low-cost directional range sensors, without reliance on dense sensing modalities such as LiDAR or vision-based systems. The behavior grid map dynamically aggregates sparse measurements into a structured local representation that supports real-time decision-making for obstacle avoidance and target reaching. Extensive simulations across environments of varying sizes and obstacle densities demonstrate that the proposed standard and enhanced methods achieve higher success rates than PPO variants and Model Predictive Control (MPC) (94\% vs. 79--90\% in small-scale high-congestion scenarios, and 83\% vs. 62--71\% in large-scale high-congestion scenarios), while maintaining real-time performance. Real-world experiments across four scenarios further confirm practical feasibility, with consistent target-reaching behaviour and no collisions under the tested conditions.
Mohammad Reza Ranjbar Divkoti, A. Aguiar· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
