Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions, allowing the drafter to learn from target feedback on both accepted and rejected proposals, focusing on the draft-induced errors that limit speculative acceptance.
Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both ef...
Yi-Zhuo Li, Jian-Hao Yan, Yun Luo et al.· 1 citation
This work introduces a student reference KL loss and mask the advantages of special termination tokens to mitigate the problem of excessive generation length and frequent truncation, and improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the...
Hao-Nan He, Hao Lei, Yun Luo et al.· 5 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.