Prompted to Explore: Training-Time-Only LLM Proposals for Sample-Efficient Pushing Grasping Policies

Abstract

Learning long-horizon robot manipulation remains difficult and time-consuming, especially under sparse rewards due to inefficient exploration and reward assignment. We present a minimal integration of large language models (LLMs) with reinforcement learning (RL) in which the LLM is used strictly as an online action proposer during early training to help with the RL agent. Given a task description and a compact scene abstraction, the LLM outputs a single parameterised primitive (pushing or grasping) that is validated by lightweight safety checks and mixed with policy actions via an epsilon-greedy scheduler. Besides, we study two simple policies that preserve the same low-level learner (grasping-biased model): (i) pushing-first reward shaping that temporarily increases the relative reward of pushing early in an episode, and (ii) a Proximal Policy Optimization (PPO) high-level switch that explicitly selects between push and grasp while Soft Actor-Critic (SAC) handles low-level parameterisation. In simulation, LLM-guided exploration accelerates early learning, and the proposed methods help further mitigate grasping overuse in the cluttered environment. Besides, the LLM-guided RL agent could achieve a higher success rate at 89.5% compared with the baseline method at 68%.