Zhen Huang

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example, is proposed and effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

Diagnosing and Mitigating Context Rot in Long-horizon Search

Through a systematic study of four flagship models, a previously overlooked phenomenon is identified: under extensive context, models give up or provide uncertain incorrect answers long before exhausting the context window.

SciPedia: Unlocking the Value of Scientific Data for Pre-training