Masaru Isonuma

Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

This work proposes a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target, and defines an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run.