We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, d...
Alexander Stern, Tim Rocktäschel, Acyr F. Locatelli et al.· 0 citations
A maximum-entropy Iterative Proportional Fitting framework is used that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.
Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh et al.· 0 citations
This work operationalizes the importance of a reasoning step as its advantage: the change in expected reward, e.g., producing the correct final answer, from including that step, estimated via Monte Carlo rollouts, and evaluates whether LLM judges can identify high-advantage steps and finds that sufficiently capable LLM...
Kevin Du, A. Hoyle, L. Ruis et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.