From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasoners
TL;DR
This work studies whether learning from both the controller's decisions and the experts'reasoning and code improves its generation after expert removal, and introduces Rivet for collaboration internalization, supporting the effectiveness of training on the content and structure of verified collaborations.
Abstract
A compact controller can coordinate stronger experts by selecting whom to consult, formulating requests, and integrating their responses. We study whether learning from both the controller's decisions and the experts'reasoning and code improves its generation after expert removal. We introduce \textsc{Rivet} for \emph{collaboration internalization}: expert-augmented reinforcement learning applies a shared outcome signal to controller decisions and returned expert spans, and verified trajectory internalization consolidates complete successful interactions through format-aware supervised training. The deployed controller generates reasoning, code, and interaction structure with local Python execution and no external LLM. Across seven competition-mathematics benchmarks, RIVET-1.7B and RIVET-4B achieve average accuracies of $28.25\%$ and $44.16\%$; Stage~II improves RIVET-4B's accuracy after expert removal by $6.49$ points, and GPQA-Diamond results provide evidence of generalization to scientific reasoning. Ablations show gains from ordinary trajectory supervision and additional format weighting, supporting the effectiveness of training on the content and structure of verified collaborations.
