RM-OPD: On-Policy Distillation for Generative Reward Models
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly o...
Qing-Lin Su, Li-Cheng Pan, Zhen Fang et al.
· 0 citations