Michael Y Hu

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time

OP-Mix (On-Policy Mix), a data mixing algorithm that operates across pretraining, midtraining, and instruction tuning, is introduced, suggesting a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data.