CodeRM-NT: Reward Model for Code RL without Unit Tests

Abstract

Providing accurate reward signals for code generated by large language models (LLMs) is a significant challenge in applying reinforcement learning (RL) to code generation. Existing methods rely on unit tests to evaluate code correctness and provide rewards, which are hindered by the difficulty of acquiring and verifying reliable unit tests at scale. In this work, we propose C ODE RM-NT, a code reward model with no reliance on unit tests. Our method leverages Monte Carlo Tree Search guided by LLMs to generate code snippets and judges execution traces to annotate code with reward signals. We use the rewards to train C ODE RM-NT that is capable of providing rewards for code during RL. C ODE RM-NT also facilitates curriculum learning by scoring and sorting training samples based on their difficulty. Experimental re-sults demonstrate that training with C ODE RM-NT consistently outperforms synthetic unit test-based rewards, yielding superior performance on multiple code generation benchmarks. Additionally, curriculum learning based on C ODE RM-NT further enhances model performance. Our code and dataset are available at: https://github.com/THUDM/CodeRM-NT .