Quartz Optimizer: Robust Gradient Shaping and Bounded Adaptive Steps for Stable Deep Learning Training

Abstract

Optimization plays a critical role in training deep neural networks, directly impacting convergence speed, model generalization, and stability. While existing methods such as stochastic gradient descent (SGD) and adaptive optimizers like Adam and AdamW have achieved significant success, they exhibit limitations in handling extreme gradients and noisy updates, and maintaining stable convergence across diverse architectures and datasets. In this study, we propose Quartz, a novel optimizer that combines momentum accumulation with bounded adaptive scaling to improve convergence efficiency and robustness. Quartz introduces a gradient-saturation mechanism that prevents excessively large updates and enforces adaptive step size bounds, thereby addressing the key limitations observed with conventional optimizers. We evaluate Quartz across three benchmark datasets (MNIST, Fashion-MNIST, and Arabic Character Classification) using multiple convolutional neural network (CNN) architectures, including GoogLeNet, VGGNet, and ResNet-18. The experimental results demonstrate that Quartz achieves up to 99.54% test accuracy on MNIST, 91.6% test accuracy on Fashion-MNIST, and 98.27% test accuracy on Arabic Character Classification. It consistently outperformed or matched the results achieved using state-of-the-art adaptive optimizers under identical training conditions. In terms of efficiency, Quartz reduces training time in several settings while maintaining higher accuracy. Across all the experiments, Quartz also shows statistically significant improvements (p< 0.001 in most comparisons) and smoother convergence behavior, indicating improved optimization stability. These findings highlight Quartz’s potential as a reliable and efficient tool for use in optimization strategies for a broad range of deep learning tasks.