Optimization plays a critical role in training deep neural networks, directly impacting convergence speed, model generalization, and stability. While existing methods such as stochastic gradient descent (SGD) and adaptive optimizers like Adam and AdamW have achieved significant success, they exhibit limitations in handling extreme gradients and noisy updates, and maintaining stable convergence across diverse architectures and datasets. In this study, we propose Quartz, a novel optimizer that combines momentum accumulation with bounded adaptive scaling to improve convergence efficiency and robustness. Quartz introduces a gradient-saturation mechanism that prevents excessively large updates and enforces adaptive step size bounds, thereby addressing the key limitations observed with conventional optimizers. We evaluate Quartz across three benchmark datasets (MNIST, Fashion-MNIST, and Arabic Character Classification) using multiple convolutional neural network (CNN) architectures, including GoogLeNet, VGGNet, and ResNet-18. The experimental results demonstrate that Quartz achieves up to 99.54% test accuracy on MNIST, 91.6% test accuracy on Fashion-MNIST, and 98.27% test accuracy on Arabic Character Classification. It consistently outperformed or matched the results achieved using state-of-the-art adaptive optimizers under identical training conditions. In terms of efficiency, Quartz reduces training time in several settings while maintaining higher accuracy. Across all the experiments, Quartz also shows statistically significant improvements (p< 0.001 in most comparisons) and smoother convergence behavior, indicating improved optimization stability. These findings highlight Quartz’s potential as a reliable and efficient tool for use in optimization strategies for a broad range of deep learning tasks.
Ahmad Raza Khan, Sarab Almuhaideb· Electronics· 0 citations
Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Sarab Almuhaideb, Ahmad Raza Khan· Applied Sciences· 0 citations