Ruben Belo

AdamX: Cosine similarity meets gradient descent

AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes is introduced, which is scalable, model-agnostic, and straightforward to integrate into existing training pipelines.