The Gradient Descent Based Adaptive Optimizer Algorithm in Deep Learning

Abstract

Gradient descent-based optimization algorithms are widely used in deep learning, where adaptive optimizers dynamically adjust parameter updates to improve training efficiency. However, most existing adaptive optimizers mainly rely on historical gradients while overlooking short-term gradient fluctuations. Although methods such as diffGrad and AngularGrad attempt to address this issue through gradient-difference or angle-based correction mechanisms, their performance improvements are often task-dependent, and some approaches introduce additional computational overhead due to complex vector operations. To address these limitations, this paper proposes a lightweight short-term gradient-aware mechanism based on gradient sign consistency, which improves optimization stability while maintaining low computational complexity. Based on this idea, two Adam-based optimizers, sigSignGrad and tanhSignGrad, are developed by incorporating adaptive friction coefficients constructed through Sigmoid and Tanh mappings, respectively. In addition, the proposed friction coefficient can be flexibly integrated into the Adam optimizer family as a lightweight plug-in module. Extensive experiments are conducted across multiple computer vision and natural language processing tasks, including image classification, object detection, semantic segmentation, machine translation, language modeling, and language understanding benchmarks. To evaluate the robustness and generalization capability of the proposed methods, multiple datasets and model architectures are considered within the same task settings, covering both training-from-scratch and pre-trained fine-tuning scenarios. Experimental results show that sigSignGrad achieves the best overall performance on multiple visual tasks, including all evaluated Vision Transformer benchmarks, Pascal VOC object detection, and semantic segmentation datasets. In natural language processing tasks, tanhSignGrad achieves the best BLEU scores on the IWSLT14 De-En translation task and consistently outperforms existing optimizers in language modeling experiments under both pre-trained and training-from-scratch settings. Furthermore, the proposed methods maintain computational efficiency comparable to diffGrad while providing more stable optimization behavior across diverse architectures and datasets.