The Consistency Optimizer Family: Modulating Parameter Updates via Local Trajectory Alignment

Abstract

First-order gradient-based optimization is fundamental to training neural networks, yet standard adaptive and momentum-based methods primarily scale parameter updates using historical gradient magnitudes. As a result, they do not explicitly leverage the directional agreement between the current gradient and the accumulated momentum, which can limit their ability to accelerate along stable optimization directions while suppressing noisy oscillations. We introduce the Consistency Optimizer Family, comprising ConsistencySGD and ConsistencyAdamW, which modulate parameter updates based on the alignment between the current gradient and the momentum direction. The method computes a bounded consistency metric that measures the normalized directional agreement between these two signals. Updates are amplified when gradients consistently follow the optimization trajectory and dampened when directions conflict. The proposed algorithms are simple to implement, require negligible additional memory, and introduce only one additional hyperparameter. Empirical evaluations across multiple domains demonstrate consistent improvements over standard optimizers. On tabular regression (California Housing), ConsistencySGD achieves a 1.32% relative MSE reduction over SGD-M and 2.57% over AdamW. On IMDB sentiment classification, ConsistencyAdamW improves accuracy by 0.65% over SGD-M and 0.52% over AdamW. On CIFAR-10 image classification, ConsistencySGD improves accuracy by 1.28% over AdamW while matching the performance of SGD-M. These results suggest that explicitly incorporating gradient–momentum alignment into update scaling provides a simple and effective mechanism for improving optimization stability across diverse learning tasks.