Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
Activation-Keyed Momentum (AK-Momentum), which builds direction-awareness into the momentum update rule, is proposed, and it is proved that it is a valid momentum, that it applies the input-side curvature correction without matrix inversion, and that it clears stale directions faster than EMA under both a fixed and a d...