Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude

Abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning because it provides stable on-policy learning with relatively simple implementation. However, standard PPO uses a fixed clipping threshold, applying the same trust-region width to all training samples even though the reliability of policy updates may vary across states and actions. This paper studies a family of adaptive-clipping PPO variants that replace the constant clipping parameter with sample-dependent thresholds derived from policy and return statistics. We introduce Dual-Adaptive Z-Score PPO (PPO-DAZ3), which dynamically adjusts the clipping interval using two signals: action confidence, derived from the probability of the selected action under the current policy, and update confidence, measured from normalized advantage magnitude. This formulation enables persample adaptation while preserving the standard PPO training pipeline. Experiments on three classic reinforcement learning benchmarks-LunarLander-v3, CartPole-v1, and Acrobotv1-show that adaptive clipping can improve learning performance in some environments. PPO-DAZ3 achieved the strongest results on LunarLander-v3, while the smooth advantage-based variant PPO-AZ3S performed best on Acrobotv1. On CartPole-v1, standard PPO remained competitive, indicating that adaptive clipping is most beneficial on more challenging control tasks.