Reward-Quality Divergence (RQD): Code, Saved Runs and Analysis Scripts (Anonymous Review Artifact)

Abstract

Anonymous review artifact for the paper Reward-Quality Divergence: Telling Judge Gaming from Judge Error in Visual RL. Please do not try to identify the authors. RQD measures the gap between a judge's reward and an anchor-based quality score during reinforcement learning, subtracts the gap that exists before training (the floor), and labels the remaining gap growth as gaming, drift, collapse or none. Contents Code: the measure, the anchor checker, the judge wrapper with its prompt, the control rewards, the second grader and the re-scoring job. Results: the saved answers, rewards and anchor scores of all 154 training runs reported in the paper, plus the re-scoring and second-grader results. Analysis scripts: CPU-only scripts that rebuild every table of the paper from the saved results in a few minutes. Data: item files with the question, reference answer and anchor fact of every item (no images; the images come from the public ChartQA, GQA and DocVQA datasets). Human labels: the 100 human correctness labels used to check the anchor. Checks included: one script recomputes the anchor score of all 219,500 saved answers from the item files and the released checker and compares it with the stored score; unit tests cover the measure, the anchor and the rewards. The GRPO training loop and the cluster job scripts are not part of this review artifact; they will be released with the published paper. All training settings are listed in the paper's appendix. Code: MIT license. Result files and human labels: CC BY 4.0. The datasets keep their original licenses. See README.md for the commands.