VBON: A Relational Action-Graph Network for Mitigating Path Fixation in Sparse-Reward Reinforcement Learning.

Abstract

Deep Reinforcement Learning (DRL) is often hindered by significant exploration challenges in dynamic environments with sparse rewards. A critical failure mode in such settings is Path Fixation, where the agent prematurely converges to the first discovered yet suboptimal solution trajectory. This convergence limited further exploration for more robust or efficient alternatives. To address this fundamental issue, we introduce the Value-Based Optimization Network (VBON), a novel neural learning framework that enhances policy learning through relational modeling of action sequences. The core of VBON is a Graph Convolutional Network (GCN) that operates on a dynamically constructed graph of the agent's own action trajectory. By explicitly modeling the spatio-temporal dependencies within this sequence, VBON learns to reason over trajectories by capturing their global quality. This trajectory-level understanding is then used to generate a dense and structured intrinsic reward that guides the policy out of a local optima region and promotes more efficient exploration. We evaluate VBON on a challenging high-dimensional Uncrewed Ground Vehicle (UGV) navigation benchmark, where environments are highly dynamic and reward signals are extremely sparse. And extensive experiments validate that VBON significantly outperforms state-of-the-art model-free and model-based methods in both sample efficiency and final policy performance.