It is shown that any discounted-reward problem can be recast as a modified, reward-free problem with a single success state and an equivalent optimal policy, and any value-learning problem can be trained by classification with cross-entropy, in a way that is in principle exact and suffers no loss of precision.
A principled reward decomposition framework is introduced that decouples verifiability from sparsity, and two new reward functions are proposed: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic imp...
Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets. Although scalar MSE is statistically valid for estimating the conditional expected return, sparse binary rewards in reinforcement learning with verifiable rewards (RLV...
Zhi-Jian Zhou, Long Li, Xuan Zhang et al.· 2 citations· ⚡2
This work proposes a game-theoretic framework that gives this reward-retention trade-off an explicit statistical interpretation, and provides a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning p...
Keegan Harris, Brian Lee, Ian Waudby-Smith et al.· arXiv.org· 0 citations
Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation, and dynamically reallocates optimization effort toward under-optimized obje...
Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training.
Priyank Agrawal, Ankur Samanta, S. Ghasemlou et al.· arXiv.org· 1 citation
Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock ov...
Le-Qi Zheng, Jin-Bo Su, Fang Niu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
