Wasserstein policy gradient (WPG) reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance and it is proved that this ODE is globally well posed and converges exponentially from every admissible initialization.
Abstract
Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form $\exp(-c/\tau)$, while retaining the usual dependence on the conditioning of the control problem.
A hybrid policy combining time-independent affine feedback with finite-horizon steering over a terminal layer is constructed, proving that its worst-case cost gap decays exponentially with the terminal-layer length uniformly in the horizon, which helps reducing the computation cost for long-horizon robust controls.
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unr...
Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv· 0 citations
Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A...
Yiyang He, Zhi-Chun Zhou, Zi-Wei Wang et al.· 0 citations
This paper develops a model-free reinforcement learning (RL) algorithm based on the stochastic maximum principle for continuous-time stochastic control problems with continuous state and action spaces. For a parameterized Markovian policy, we establish the existence of the decoupling field for the adjoint backward stoc...
Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare...
Combining the policy-iteration error with a quantitative soft-max approximation bound, it is shown that the iterated policies constitute weighted $\varepsilon$-equilibria for the original unregularized problem and derive an explicit regret estimate.
Fengyu Cao, Zhen-Hua Wang· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
