A four-rung ablation ladder of increasing targeting precision applied to the entropy bonus, holding the entropy setpoint fixed and varying only where the bonus is placed answers the question: does learned, per-rollout targeting of exploration beat heuristic targeting beat no targeting?
SignBALANCE is proposed, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling.
Jiamian Wang, Samyadeep Basu, Koustava Goswami et al.· 0 citations
AdaKP is an online selector that re-chooses each problem's KP subset over the course of RL training, an entropy proxy that scores a KP by the reduction in next-token entropy it induces in a single inexpensive forward pass, with a provable bound on its truncation bias.
This work compares four reward designs that span lexical suppression, anti-refusal shaping, rubric-based broad answering, and an explicit refusal contrast, and shows that optimization success is not equivalent to behavioral unlearning: RWKU forget scores, held-out completion audits, terminal training-rollout audits, an...
The nature of test-time exploration in RLVR-trained LLMs is investigated by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence to delineate between entropy arising from stylistic variations and genuine inferential...
Soumadeep Saha, Krish Sharma, Akshay Chaturvedi et al.· 1 citation
Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training.
Priyank Agrawal, Ankur Samanta, S. Ghasemlou et al.· arXiv.org· 1 citation
Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout, is proposed and established as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.
Pei-Xuan Han, Runnan Wang, Ketan Ramaneti et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
