POLAR: Policy Optimization through Larger Reasoners

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a practical way to improve the reasoning and decision-making abilities of large language models, but its training signal is often sparse. When a policy faces prompts beyond its current ability, every sampled trajectory in a group may fail the verifier, resulting in an all-zero group. In this case, group-relative policy optimization gives little useful gradient even when the prompt contains valuable learning content. This capacity-difficulty mismatch is especially costly for smaller models used in knowledge-intensive and web-interaction tasks. We present POLAR, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner. POLAR uses the standard GRPO branch when at least one sampled response succeeds, and switches to a supervised objective over verified teacher trajectories only when the whole student group receives zero reward. The method can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle. We formalize the switching criterion, teacher selection objective, mixed RL/supervised loss, and compute overhead. We evaluate POLAR on WebShop, GAIA, and WebWalkerQA using Qwen2.5-7B-Instruct, Qwen3-4B, and Qwen3-8B policies. The results show consistent gains over matched GRPO baselines, with large improvements on hard web traversal questions for the two smaller policies. Ablations and diagnostics further show that the gains come from the proposed all-zero recovery mechanism.