Generative foundation models are attracting interest for their ability to produce desired outputs from demonstrations given at inference time, without updating parameters. However, since a few demonstrations cannot uniquely identify the intended task, the challenge is how to learn and sample from an output distribution...
Guoji Fu, Tomoya Wakayama, Ryotaro Kawata et al.· 0 citations
This paper proposes the concept of non-hackable confidence reward schemes and defines a spectrum of such reward schemes for RL confidence calibration training in LLMs, and demonstrates that selective confidence reward hacking can occur in practical datasets with reward schemes that are not designed to be non-hackable.
C. Tan, Zhuoyi Lin, M. Motani et al.· arXiv.org· 0 citations
Best Practice Critic Optimization (BPCO) is developed, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation and shows that a carefully designed critic provides a reliable alternative to...