Skip to content
Open access

Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement

Nov 2026 · Symmetry · 0 citations

Abstract

Offline reinforcement learning aims to learn effective policies from fixed datasets without further interaction with the environment. However, limited behavior-policy coverage can lead to value-estimation errors and unreliable policy improvement, particularly when multiple critics provide conflicting update directions. To address these challenges, we propose Probability-Ratio-Constrained Q-Learning (PCR-QL), which integrates behavior-density modeling, conservative ensemble-based Q-learning, and probability-ratio-regularized policy extraction to control distributional shift. Building on this framework, we further introduce Consensus-Gated PCR-QL (CG-PCR-QL), which evaluates critic-specific actor gradients using shared states and policy noise and adaptively attenuates the ensemble-mean policy-improvement direction according to the weakest critic-gradient agreement. Theoretically, we establish results connecting ratio-constrained Bellman approximation, behavior-density estimation errors, policy-feasibility repair, and multi-critic gradient geometry, together with conditional guarantees for completed optimizer updates. Experiments on eleven Datasets for Deep Data-Driven Reinforcement Learning (D4RL) benchmark tasks compare nine policy-extraction rules across ten matched runs. CG-PCR-QL achieves a mean performance improvement of +6.32 (95% CI [5.14, 7.50]) over PCR-QL and a pooled interquartile-mean (IQM) improvement of +7.19 [5.63, 8.69]. Furthermore, an analysis of 2400 paired Adam-update events shows an increase of +13.0 [11.1, 14.9] percentage points in the frequency of simultaneous improvement across all critics. Additional ablations and diagnostics examine the effects of gradient consensus, behavior-density estimation, parameter sensitivity, and computational cost. These results highlight the effectiveness of integrating probability-ratio constraints with critic-gradient consensus for more reliable policy improvement in offline reinforcement learning.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.