Jul 2026· Anusandhanvallari· pp. 4838-4848· 0 citations· 24 references
TL;DR
A phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD is proposed that provides a simple and effective solution for handling non-convex optimization problems in deep learning.
Abstract
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on one benchmark dataset, MNIST dataset, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems in deep learning.
This paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks that achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks.
Weiwei Guo· Applied and Computational En...· 0 citations
Deep neural networks are commonly trained using adaptive optimization methods such as Adam because they converge quickly and perform well under stochastic training conditions. Despite these advantages, recent research has revealed several important drawbacks of Adam. In particular, the optimizer tends to converge toward sharp minima, exhibits oscillatory behavior during loss optimization, and can produce unstable parameter updates when training in highly noisy environments. In this work, we propose NeuroFuzzyAdam (NF Adam), a novel optimizer that integrates fuzzy logic-based scalar modulation directly into the Adam update rule to improve learning stability and generalization. By introducing a bounded correction term via hyperbolic tangent transformations, NF Adam adaptively regulates step sizes based on moment estimates while preserving Adam’s key strengths. We provide theoretical convergence guarantees under convexity and bounded-gradient assumptions, matching Adam’s O(1/T) regret bound. Empirically, we evaluate the NF Adam across both image (MNIST, Fashion-MNIST, CIFAR-10) and tabular (UCI Iris, Wine, Breast Cancer) datasets using consistent architectures. Results show that NF Adam improves training smoothness, reduces gradient norm variance, and achieves competitive classification accuracy compared to Adam, SGD, RMSprop, and AdamW. These findings suggest that incorporating fuzzy logic into optimization dynamics presents a promising direction for enhancing robustness in neural network training, especially in scenarios with high gradient noise or non-smooth loss surfaces.
Susilo Hariyanto, Siti Khabibah, Retno Putri Dwi Rahmawati et al.· WSEAS Transactions on System...· 0 citations
Optimization plays a critical role in training deep neural networks, directly impacting convergence speed, model generalization, and stability. While existing methods such as stochastic gradient descent (SGD) and adaptive optimizers like Adam and AdamW have achieved significant success, they exhibit limitations in handling extreme gradients and noisy updates, and maintaining stable convergence across diverse architectures and datasets. In this study, we propose Quartz, a novel optimizer that combines momentum accumulation with bounded adaptive scaling to improve convergence efficiency and robustness. Quartz introduces a gradient-saturation mechanism that prevents excessively large updates and enforces adaptive step size bounds, thereby addressing the key limitations observed with conventional optimizers. We evaluate Quartz across three benchmark datasets (MNIST, Fashion-MNIST, and Arabic Character Classification) using multiple convolutional neural network (CNN) architectures, including GoogLeNet, VGGNet, and ResNet-18. The experimental results demonstrate that Quartz achieves up to 99.54% test accuracy on MNIST, 91.6% test accuracy on Fashion-MNIST, and 98.27% test accuracy on Arabic Character Classification. It consistently outperformed or matched the results achieved using state-of-the-art adaptive optimizers under identical training conditions. In terms of efficiency, Quartz reduces training time in several settings while maintaining higher accuracy. Across all the experiments, Quartz also shows statistically significant improvements (p< 0.001 in most comparisons) and smoother convergence behavior, indicating improved optimization stability. These findings highlight Quartz’s potential as a reliable and efficient tool for use in optimization strategies for a broad range of deep learning tasks.
Ahmad Raza Khan, Sarab Almuhaideb· Electronics· 0 citations
AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves, and achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain.
A. K. Pandey, Umang Chaturvedi, Aatish Rana et al.· 0 citations
This work provides a rigorous error analysis for the momentum SGD optimizer and establishes convergence rates for the momentum optimizer in terms of the size of the learning rate, the size of the mini-batch, and the size of the one-point convexity constant.
