When do data mixtures improve scaling laws? Insights from high-dimensional regression
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely...
Di-Yuan Wu, Le-Han Chen, Theodor Misiakiewicz et al.
· 0 citations