LocalProp: Neuro-Localized Memory-Efficient Backpropagation
This work proposes LocalProp, a training procedure that locally updates the weights of a model, where the pre-training is based on I-JEPA, and empirically shows that it reaches good performance at a fraction of GPU peak memory.