Latent MeanFlow Policy Optimization for Offline Reinforcement Learning.
TL;DR
LaMPO is proposed, a generative policy framework that formulates offline RL as latent generative policy optimization as well as a behavior-conditioned latent distribution, which provides an informative prior and reduces the modeling burden of the generative process.
Abstract
Offline reinforcement learning (RL) aims to derive effective policies from fixed datasets without environment interaction. While generative models such as diffusion and flow matching improve policy expressiveness, they often suffer from high computational cost due to multi-step sampling and limited representational richness from uninformative Gaussian priors. To address these challenges, we propose Latent MeanFlow Policy Optimization (LaMPO), a generative policy framework that formulates offline RL as latent generative policy optimization. LaMPO learns a behavior-conditioned latent distribution, which provides an informative prior and reduces the modeling burden of the generative process. Conditioned on this prior, the MeanFlow policy is trained to predict the average action velocity, enabling high-fidelity one-step action generation that avoids the cost of iterative sampling. Guided by the MeanFlow policy, the target policy is then optimized under a MeanFlow-induced behavior constraint, achieving effective policy improvement. We further establish a theoretical performance lower bound of the target policy relative to the MeanFlow policy. Extensive evaluations across 71 tasks in OGBench, D4RL, and real-world humanoid manipulation demonstrate that LaMPO achieves superior performance with high efficiency, yielding a 19% average improvement over existing state-of-the-art methods and an 81% success rate in real-world robotic tasks.