In the sparse-reward, goal-directed setting, the sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.
Abstract
We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the low-level dynamics that are sufficient to make parallel HRL learnable. When these conditions hold, we prove that HBPI-UCRL has a polynomial sample complexity in the problem parameters. In the sparse-reward, goal-directed setting, our sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.
High-level ReLearning is proposed, a novel HRL approach to address the problem of non-stationary environments at the high level since the low-level policy is constantly changing and a student network is introduced to escape the influence of past data.
A novel HRL model is proposed that supports direct off-policy correction based on a Flow-based Deep Generative Model (FDGM) that leverages the inverse operation of FDGM to achieve goals aligned with the current knowledge of the lower-level policy.
Jaeyoon Kim, Junyu Xuan, C. Liang et al.· Journal of Artificial Intell...· 0 citations
Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent, is introduced and it is proved that it has the same unique optimal solution as the original PMD objective.
Zhuo-Qing Song, Hao-Tian Xu, Xi-Kun Zhang et al.· 0 citations
A unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification for best policy identification is proposed, including both instance-dependent and independent upper bounds on regret and sub-optimality gap.
This work formalizes the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and shows that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate.
Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba· 1 citation
HRL-SAC effectively addresses sparse-reward long-horizon search problems characterized by delayed rewards and continuous control, and its outperforming the flat SAC baseline reinforcement learning in terms of success rates, coverage efficiency, and convergence.
Zahra Abdalla Elashaal, Afef Hfaiedh, N. Khraief et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
