This work formalizes the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and shows that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate.
Abstract
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
A novel policy gradient method is introduced, Hindsight Policy Optimization (HPO), that projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them.
Zishang Jiang, Tingyun Li, Jinyi Han et al.· 0 citations
This work provides a first look at the AgenticAI-Supervisor platform's core capabilities through a Customer Support Agent case study demonstrating a consistent closed-loop feedback for model optimization, and mitigates reward hacking through rigorous internal state validation and testing.
Akshay Arora, Ishan Nigam, Ashutosh Aggarwal et al.· 0 citations
This work proposes Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout to ensure stable and unbiased learning from experience-augmented rollouts.
This work proposes a new quantitative semantics for STL having several desirable properties, making it suitable for reward generation, and establishes the new semantics to be the most suitable for synthesizing feedback controllers for complex continuous dynamical systems through reinforcement learning.
Nikhil Singh, Indranil Saha· Journal of Artificial Intell...· 0 citations
BPO is instantiate as Branching Policy Optimization (BPO), a sandbox-native RL algorithm that adaptively snapshots the sandbox at high-entropy decision points along a backbone trajectory, and proves this estimator is unbiased and has strictly lower variance than the trajectory-level baseline, with the reduction equal to the prefix-explained portion of return variance.
Bowei He, Yankai Chen, Xiaokun Zhang et al.· 1 citation
Reinforcement learning holds significant potential for training large language models to handle multi-turn interactive tasks, but directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback.
Qiang Liu, Taian Guo, Ruizhi Qiao et al.· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
