DynBudget: Adaptive Risk Calibration and Dynamic Budgeting for Safe Reinforcement Learning Under Distribution Drift
TL;DR
DynBudget, a closed-loop Safe RL framework integrating a learned safety critic, temperature-calibrated risk estimation, and a dynamic safety budget, is proposed and it is shown that shielding with dynamic budgets is an interpretable and viable approach to Safe RL in autonomous systems.
Abstract
Ensuring safety constraint satisfaction in non-stationary Reinforcement Learning (RL) environments remains a major challenge, as distribution drift can quickly invalidate previously calibrated risk estimates. To address this issue, we propose DynBudget, a closed-loop Safe RL framework integrating a learned safety critic, temperature-calibrated risk estimation, and a dynamic safety budget. By jointly adjusting the calibration temperature and the intervention threshold based on real-time violation feedback, the safety shield enables rapid adaptation to environmental drifts. We carried out experiments in two stages to fully evaluate the adaptation mechanism as well as the practical robustness of DynBudget. First, we studied the framework in a number of discrete MiniGrid contexts, and then we tested it on various continuous robotic tasks in Safety-Gymnasium. Our empirical results indicate that our approach is able to effectively reduce safety violations, shorten recovery time after a drift, and preserve competitive task performance when faced with abrupt changes in hazards or physical perturbations. Finally, we show that shielding with dynamic budgets is an interpretable and viable approach to Safe RL in autonomous systems.