Skip to content

GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control

Sep 2026 · arXiv.org · Vol abs/2609.19850 · 0 citations · 23 references
Computer Science

TL;DR

Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control, is proposed and significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods.

Abstract

Actor-critic architecture has been widely used in continuous robot control. However, they rely on learning a value network, introducing additional computational overhead during training. Moreover, policy learning may also be affected by the approximation error of value estimation. Critic-free group relative policy optimization methods provide a simpler training approach by removing the need for a critic. However, they fail to learn long-term action outcomes when directly applying immediate rewards to policy optimization in dense-reward environments. To address these problems, we propose Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control. GR2PO estimates the discounted returns from the parallelly collected trajectories, performs group normalization at each rollout time index, and uses relative advantages and clipped targets to update the policy. To evaluate the effectiveness of the proposed framework, we instantiate it on robot control simulation environments and deploy the model to a real-world edge device. The results show that GR2PO significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods. Furthermore, GR2PO demonstrates competitive training efficiency. Inference tests on NVIDIA Jetson TX2 demonstrate the feasibility of deploying the learned policies on edge platforms. Further ablation experiments analyze the effects of parallel group size, return estimation methods, and target clipping ratio on learning performance. To support follow-up research, we will make the complete code publicly available after the paper is accepted, including the framework implementation, experimental configuration, and training and evaluation scripts.

View source

Similar papers

Preprint Aug 2026

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

A unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA) is introduced that enables multiple actors to share a centralized multi-head critic and substantially improves both sample efficiency and policy performance.

Changhao Li, Yifang Zhang, Heng Zhang et al. · 0 citations
Sep 2026

Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models

This work introduces PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation, and demonstrates zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2.

Kowndinya Boyalakuntla, Yu-Han Liu, Abdeslam Boularias · 0 citations
#artificial intelligence Preprint Oct 2026

Optimal Control with Learned Critics under Unmodeled State Dependencies

A learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia and is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynami...

P. Schoch, Markus Ryll · 0 citations

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias throu...

Hsiao-Ru Pan, Florent Draye, Bernhard Scholkopf · 0 citations
Sep 2026

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

A framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions is proposed.

Emek Barış Küçüktabak, Karankumar Patel, Zhao-Dong Yang et al. · 3 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.