When Optimization Forgets the Minority: Bias Amplification in Policy-Gradient Reinforcement Learning

Abstract

Bias amplification, the tendency of trained models to be more biased than the data that produced them, is well documented in supervised learning. We study it in policy-gradient reinforcement learning, where there is no static dataset: each episode belongs to a latent group that shapes the reward but is invisible to the policy. Under exact gradients the problem reduces to an averaged Markov decision process, so imbalance acts only through the weights of the averaged reward; the groups reappear in the variance of sampled gradients, and state visitation, which the policy controls, amplifies the effect of the weights. We separate the disparity induced by the problem from the disparity induced by the optimization. At the destination, with a full-support start distribution, unregularized softmax policy gradient drives to zero, at an explicit rate, the probability of every action that is suboptimal for the averaged problem; under strong imbalance these include the minority-preferred actions wherever the groups conflict. Entropy regularization restores a floor with closed-form thresholds in the bandit, where the regularized optimum is the ρ-average of the per-group optima. On the journey, near-deterministic policies are slow to leave (an action well), and in Markov decision processes the visitation distribution adds a second slowdown (a visitation well). In a gridworld where both groups share the same optimal policy, exact policy gradient from a fixed start state locks the minority out (its return stays 47% below optimal after 3·10⁴ iterations), sampled REINFORCE never finds the common optimum in 270 runs, and, with full state coverage, the time to recover the minority's goal grows faster than 1/(1 − ρ). Bias can thus be a property of the optimization path rather than of the data or of the optimum.