Cross-Architecture Assessment of Hyperparameter Optimization Techniques in Convolutional Neural Networks
TL;DR
It is demonstrated that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Abstract
Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.