TL;DR
A novel pruning method which uses the oscillations around 0 that a weight has undergone during training in order to determine its saliency and achieves state-of-the-art performance for levels of sparsity of 99 .
AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes is introduced, which is scalable, model-agnostic, and straightforward to integrate into existing training pipelines.
Across CIFAR-10, CIFAR-100, and TinyImageNet with ResNet and vision transformer backbones, lifted training improves test accuracy by up to five percentage points over variants without lifting under a shared tuning protocol.
Evaluating deep learning optimizer is challenging because confounding factors such as hyperparameter tuning and optimizer-model co-design often obscure true algorithmic performance. We present results of the updated AlgoPerf V1 benchmark, in which we evaluate a new set of training algorithms on wall-clock time-to-targe...
Learning an ensemble of GFlowNets to sample from a discrete target distribution has become a common approach for achieving better state space exploration and convergence than that of a monolithic sampler. However, these methods often add a substantial runtime overhead to the base model, and their conceptual connection...
This work derives the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers in networks with sine activations, and is the first work to use the sine function's periodic symmetry.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
