Skip to content

CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning

Sep 2026 · arXiv.org · Vol abs/2609.08853 · 1 citation · 33 references
Computer Science

TL;DR

This work presents CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy.

Abstract

Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.

View source

Similar papers

GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planning

GEM-MPC is proposed, an MPPI-based reinforcement learning method that improves the interaction between planning and learning and selectively learns from stored planning distributions only when they provide a better target than the current prior, reducing the impact of stale planning data without requiring full reanalys...

Álvaro Serra-Gómez, T. Moerland · 0 citations
Sep 2026

Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models

This work introduces PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation, and demonstrates zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2.

Kowndinya Boyalakuntla, Yu-Han Liu, Abdeslam Boularias · 0 citations
#artificial intelligence Preprint Oct 2026

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning

With massively parallel simulation, on-policy Reinforcement Learning methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a...

D. Affinita, Ming-Jing Xu, Rudolf Reiter et al. · 0 citations

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for embodied intelligence, but fine-tuning them with reinforcement learning (RL) remains constrained by the cost of real-world robot interaction. Model-based reinforcement learning (MBRL) reduces this cost by using a learned world model to generate...

Yi-Fei Sheng, Hao-Xiang Ren, Zhi-Long Zhang et al. · 0 citations
Conference Open access Sep 2026

Reinforcement Learning: A Comprehensive Survey of Foundational Algorithms, Recent Advances, and Future Directions

Reinforcement Learning (RL) has emerged to be one of the most radical ideas in artificial intelligence since it enables the learner to learn the desired behavior through a response to the environment. The survey provides the overall picture of reinforcement learning, its development history beginning with the classical...

N. Ramshankar, D.Elangovan, S. S et al. · 0 citations

Refinement-Based Flow Policy Optimization

Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse, and across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task.

B. Park, Hyukjun Yang, Donghwan Lee · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.