Skip to content

Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control

Sep 2026 · 0 citations · 46 references
Computer Science

TL;DR

Sampling-Guided Policy Search (SGPS), which couples recurring action-target refinement by sampling-based model-predictive control with first-order policy optimization with first-order policy optimization, and shows that refinement improves policy learning beyond initialization and tracking alone.

Abstract

Learning visual policies for locomotion and manipulation requires coordinating contact with the environment and can incur substantial computation and GPU memory costs. First-order policy gradients (FoPG) reduce training cost through differentiable simulation, but local optimization can converge to unintended contact patterns. To address this shortfall, we propose Sampling-Guided Policy Search (SGPS), which couples recurring action-target refinement by sampling-based model-predictive control with first-order policy optimization. Behavior cloning initializes the policy from sampled actions; training then alternates sampling-based refinement with short-horizon FoPG updates under perturbed initial states and randomized dynamics. For visual policy training, we use a decoupled FoPG formulation that excludes rendering from the computation graph, enabling direct learning from depth observations without a state-policy teacher. On a single GPU, SGPS learns policies for locomotion, obstacle traversal, crate pushing, and bimanual carrying on simulated Unitree Go2 and G1 robots. Our experiments further show that refinement improves policy learning beyond initialization and tracking alone. For hardware deployment, the distilled policy transfers zero-shot to a real Go2 and uses onboard depth to autonomously trot, crawl, clear hurdles, and switch between these behaviors.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

VAMPS: Visual and Motor Policies from Sampling-Based Planning

This work introduces Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses Model Predictive Path Integral (MPPI) control to train reusable policies without human demonstrations, and demonstrates visuomotor pick-and-place and force-aware whiteboard erasing.

Mohamed Yassine Kabouri, Pietro Noah Crestaz, Q. Nguyen et al. · 0 citations
Sep 2026

Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic

Traversing narrow gaps is challenging for autonomous quadrotors, especially when control commands come directly from high-dimensional visual observations. Existing end-to-end methods often rely on behavior cloning or full-rollout backpropagation through time (BPTT) via differentiable simulation, which can limit policy...

Nuthasith Gerdpratoom, Tian-Chen Sun, Yi-Chao Gao et al. · 0 citations
Sep 2026

Safe Real-Time Policy Steering via Noise-Space Trajectory Optimization for One-Step Generative Policies

Insight-time steering of one-step generative policies as trajectory optimization in the policy's input noise space is proposed, which improves task success and constraint satisfaction over best-of-N sampling and action projection, while comparing favorably with gradient-guided generation at lower runtime.

Qing-Yi Chen, Joseph Ruan, Zachary Kingston · 1 citation

Copper-Policy: Focus on the Representation for Robust Robot Manipulation

Copper-Policy is introduced, which learns a compact World representation with the policy rather than relying on a predefined target space and predicts future observation embeddings conditioned on task intention without reconstructing pixels through temporal joint-embedding prediction.

Ze Feng, Yi-Xu Feng, Ling-Yu Xiao et al. · 1 citation
Sep 2026

GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control

Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control, is proposed and significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods.

Pengqin Wang, Qi-Ming Zhang, Shao-Jie Shen et al. · 0 citations

GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planning

GEM-MPC is proposed, an MPPI-based reinforcement learning method that improves the interaction between planning and learning and selectively learns from stored planning distributions only when they provide a better target than the current prior, reducing the impact of stale planning data without requiring full reanalys...

Álvaro Serra-Gómez, T. Moerland · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.