TL;DR
It is demonstrated that network architecture and its coeficients can be learned together by unifying concepts of evolutionary search within a population based traditional training process.
Common practice when training Convolutional Neural Networks (CNNs) is to utilize randomly shuffled mini-batches. During this stage of the training process, two limitations are faced. Firstly the slowing down of convergence, and secondly the limited learning signal contribution due to large quantities of samples that are considered easy while training CNNs.In our work, we aim to address these inefficiencies by proposing A*-Inspired Batch Selection (A*-BS). This is a lightweight and model-agnostic training strategy that formulates mini-batch scheduling as a heuristic search problem. In this type of batch selection process each batch is treated as a node in a search space and ranked using an A*-like score by combining a loss-based difficulty measure with a reuse penalty. The two main benefits of this method are: informative gradient updates encouragement and batch diversity selection throughout the training process. The proposed method does not modify network architectures or optimization algorithms and can be seamlessly integrated into existing training pipelines. We evaluate the approach on the twelve 2D classification tasks of the MedMNIST-v2 benchmark, using a deliberately simple architecture of approximately 2.25 × 105 parameters and comparing against the ResNet-18 and ResNet-50 baselines reported by the benchmark.On half of these tasks, the lightweight model combined with A*-BS reaches higher accuracy and AUC than both ResNet baselines, with relative gains of up to 15%. Moreover, an ablation under identical CNN architecture and hyper-parameters shows that A*-BS outperforms random batch shuffling on all twelve MedMNIST tasks. Wall-clock measurements further show that the lightweight CNN with A*-BS trains substantially faster than ResNet-18 and ResNet-50 on identical hardware.Indicative in these results is that intelligent batch ordering can partially compensate for reduced architectural complexity, thus offering a computationally efficient alternative to deeper models. Furthermore, the reliability of the proposed strategy is reinforced by its strong performance with a simple CNN, even when compared to deeper and more sophisticated architectures.
Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Understanding and exploiting the training dynamics of overparameterized deep neural networks remains a central challenge in modern machine learning. Recent evidence on Neural Collapse (NC) shows that class representations and classifiers exhibit highly structured geometry, while the Tunnel Effect suggests that only a subset of layers is essential for feature extraction. We combine these two perspectives and propose an NC-inspired training framework for simplifying deep networks during training. Our method monitors representation dynamics through the Inverse Fisher Criterion, a stable and efficient proxy for the variability collapse behavior, to identify both the split point between feature extraction and classification and the training stage at which simplification becomes viable. We then replace the trailing layers with a lightweight classification head and continue training the reduced model. Experiments on image-classification benchmarks across MLP, VGG, and ResNet architectures show that the proposed method achieves substantial parameter reductions while maintaining accuracy comparable to that of the full model. Code to reproduce the experiments can be found at: https://github.com/LorenzoSciandra/NNS.
Transfer learning has become a key technique for improving the accuracy of neural networks in low-resource, low-data environments. The quantitative comparative analysis of the pre-trained models includes ResNet50, VGG16, BERT, GPT, and the baseline CNN and LSTM models. They are compared across three different application areas: computer vision, natural language processing (NLP), and medical imaging. The five benchmark datasets used were ImageNet, CIFAR-10, SST-2, IMDB, and Chest X-Ray. All experiments used the same preprocessing pipeline and evaluation metrics (accuracy, F1 score, precision, recall, and ROC-AUC). Results showed that models trained on the pre-trained data achieved consistently greater accuracy than the baselines in all domains (9-20%) and F1-score (0.09-0.16) gains. ResNet50 achieved 92% accuracy on CIFAR-10, compared to 72% for the CNN baseline, whereas BERT hit 92% on SST-2, with 80% accuracy for LSTM. VGG16 improved the accuracy of Chest X-Ray classification from 78% to 87% and reduced training time by up to 60%. There were a few instances of minor overfitting and domain mismatch, emphasizing the need for adaptive fine-tuning strategies. The results demonstrate that transfer learning significantly improves convergence speed, generalization, and computational efficiency, making it a promising approach for AI applications across domains such as healthcare, NLP, and autonomous systems.
Deep Neural Networks (DNNs) have achieved remarkable success across diverse machine learning applications, yet designing effective architectures remains a laborious, expert-driven process. Neural Architecture Search (NAS) was introduced to automate this process, with Evolutionary NAS (ENAS) emerging as one of the most effective and widely adopted NAS paradigms. This survey provides a comprehensive and systematic review of 164 ENAS studies published between 2020 and 2024, categorized according to the specific evolutionary algorithm employed as the search strategy. Unlike prior surveys—which either treat evolutionary methods at a high level or focus on general NAS pipelines—this study is, to the best of our knowledge, the first to perform an in-depth, algorithm-level breakdown of ENAS research, covering evolutionary strategies (ES), genetic algorithms (GA) and their derivatives, Particle Swarm Optimization (PSO), Differential Evolution (DE), Quantum-Inspired Evolutionary Algorithms (QIEA), and other heuristics. Our quantitative analysis reveals that ES is the dominant paradigm (45.7% of studies), followed by GA (29.9%), with annual publication counts growing 6.6-fold from 2020 to 2024—reflecting a rapidly expanding field. Key findings include: (i) surrogate-assisted methods have grown substantially in 2023--2024, significantly reducing search cost; (ii) GA-based multiobjective approaches (NSGA-II, NSGA-III) dominate Pareto-optimal architecture search; (iii) hybridization of evolutionary search with gradient descent and predictor networks is an accelerating trend; and (iv) image classification on CIFAR-10/100 and NAS-Bench-201 benchmarks accounts for the majority of evaluations, highlighting opportunities in underexplored domains. This survey also provides a critical comparison of encoding strategies (fixed-length vs. variable-length) and an analysis of search space complexity across reviewed methods. We believe this work serves as a practical guide for researchers seeking to understand, select, and advance evolutionary strategies for automated neural architecture design.
This work proposes a mechanism that bridges these two paradigms: an LLM produces a high-quality seed architecture, then decomposes it into a "slotted architecture", a scaffold with named, interchangeable module slots that automatically defines a bounded, task-specific search space for conventional NAS to explore, without manual engineering.