This work evaluates Constrained Bayesian Optimization with the primary objective of minimizing energy consumption and subject to the constraint that the generalization performance is above some threshold and demonstrates that CBO achieves lower energy consumption without compromising the predictive performance of ML models.
Abstract
Bayesian optimization (BO) is an efficient framework for optimization of black-box objectives when function evaluations are costly and gradient information is not easily accessible. BO has been successfully applied to automate the task of hyperparameter optimization (HPO) in machine learning (ML) models with the primary objective of optimizing predictive performance on held-out data. In recent years, however, with ever-growing model sizes, the energy cost associated with model training has become an important factor for ML applications. Here we evaluate Constrained Bayesian Optimization (CBO) with the primary objective of minimizing energy consumption and subject to the constraint that the generalization performance is above some threshold. We evaluate our approach on regression and classification tasks and demonstrate that CBO achieves lower energy consumption without compromising the predictive performance of ML models.
Hyperparameter selection remains a key challenge in Bayesian optimization (BO) and Bayesian active learning (AL), as model misspecification can lead to suboptimal performance, while more accurate fully Bayesian treatments typically rely on computationally expensive MCMC sampling. This paper proposes a unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies. The central idea is to replace hyperparameter sampling with a kernel ensemble and adaptive Bayesian weighting, combined with disagreement-aware acquisition strategies. Within this unified framework, we instantiate KENDO-BO for BO and KENDO-AL for Bayesian AL, demonstrating that both arise from a common self-correcting mechanism with task-specific acquisition objectives. We further extend the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure. Thorough numerical tests on synthetic and real-world benchmarks across single-objective optimization, multi-objective optimization, and active learning demonstrate that (i) KENDO-BO achieves competitive or superior optimization performance compared to state-of-the-art methods while reducing computational overhead by up to $5\times$ and (ii) KENDO-AL achieves superior predictive calibration over MCMC-based active learning baselines with up to $27\times$ speedup.
Heng Zhang, Haotian Xiang, Qin Lu et al.· 0 citations
Bayesian Optimization (BO) is widely adopted for data-efficient optimization in scientific and engineering applications, yet its computational cost is rarely evaluated alongside optimization performance. Here we present a systematic, compute-aware study of BO that evaluates surrogate models along two axes: optimization quality and computational frugality. Across eight benchmark functions and nine real-world datasets spanning materials science, mechanics, robotics, chemistry, and machine learning, we benchmark four surrogate models: Gaussian Processes, Random Forests, NGBoost, and Bayesian Adaptive Spline Surfaces. We show that Gaussian Process-based BO consistently incurs the highest time and memory overhead without delivering superior optimization or sample efficiency. In contrast, scalable alternatives achieve equal or better performance at a fraction of the computational cost. Motivated by these findings, we introduce a surrogate-recommendation framework that predicts the most suitable BO surrogate from inexpensive dataset characteristics. Together, these results establish FruBO as a reproducible, compute-aware baseline for Bayesian Optimization and provide practical guidance for surrogate selection under limited computational and experimental budgets.
P. Krokidas, C. Rekatsinas, Vassilis Sioros et al.· 1 citation
Many design tasks can be cast as black-box function optimization, enabling use of Bayesian optimization to find an ideal design with minimal number of trials. However, often we do not actually need the optimum but instead a sufficiently good solution is enough, for instance a material that is durable enough for its intended use. In most cases there are multiple satisfactory solutions, forming a superlevel set of the function, raising a key question of which one to prefer. We answer this by explaining why robustness to input perturbations that may occur when the solution is deployed is a good criterion and by introduce a Bayesian optimization method that efficiently finds satisficing solutions that are robust to maximally large perturbations. In contrast to previous works, we assume the inputs can be accurately controlled during optimization, but will be perturbed after the deployment.
Samuli Kinnunen, P. Mikkola, Antti Niskanen et al.· 0 citations
Hyperparameter selection plays a critical role in the convergence speed and performance of artificial neural networks (ANNs), yet conventional tuning methods such as grid search, random search, and Bayesian optimization often suffer from high computational cost and limited adaptability. To address these limitations, this study proposes an adaptive hyperparameter optimization framework based on the Bobcat Optimization Algorithm (BOA) for tuning the learning rate and momentum of an ANN. BOA employs a biologically inspired exploration–exploitation mechanism that dynamically adjusts hyperparameters according to training performance, enabling efficient search without relying on probabilistic surrogate models. The proposed BOA-ANN framework is evaluated on the Modified National Institute of Standards and Technology (MNIST) handwritten digit dataset using a three-layer feedforward neural network. Experimental results demonstrate that BOA-ANN achieves a test accuracy of 98.52%, a mean squared error of 0.014, and an F1-score of 98.30%, outperforming Bayesian optimization, the Secretary Algorithm, and Automated Model Compression (AMC) pruning under identical settings.
Mohammed Majid Abdulrazzaq, Nehad T. A. Ramaha, A. A. Hameed et al.· PeerJ Computer Science· 0 citations
Gradient-free post-training has emerged as a compelling alternative to gradient-based optimization for large language models (LLMs), but existing approaches remain costly. We ask whether structured search can identify a strong single expert under a modest evaluation budget. Motivated by evidence that useful weight updates lie in low-dimensional subspaces, we apply Bayesian optimization within a random linear embedding of weight space. Our method requires no backpropagation and uses a Gaussian process surrogate to guide candidate evaluations efficiently. Across several reasoning benchmarks with Qwen2.5-Instruct models from 0.5B to 3B parameters, Bayesian optimization using five times less candidate evaluations matches or exceeds RandOpt. These results show that surrogate-guided search can substantially reduce the evaluation cost of gradient-free post-training while producing stronger deployable single experts.
Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra et al.· 0 citations
Bayesian optimization is a surrogate-based global optimization method that is increasingly being used for engineering design. However, most methods are designed to be sequential, and in optimization problems where the functional evaluation can be easily parallelized, batch methods are more effective at reducing real-time computation. While existing batch multi-objective Bayesian optimization (B-MOBO) methods achieve strong performance, they are typically purpose-built from scratch, limiting ability to leverage the extensive library of proven sequential acquisition functions for parallel settings. In this paper, a generalizable transformation methodology is developed that converts compatible single- or multi-objective sequential Bayesian optimization methods into B-MOBO methods. The key innovation is a Euclidean distance-based composite acquisition function with multi-objective penalization averaging, which combines multiple sequential acquisition functions while incorporating objective-wise penalization information from all objectives and preventing redundant sampling in batch selection. To demonstrate the schema, this methodology is applied to two representative sequential acquisition strategies, the expected improvement method (single-objective) and the quality metrics method (multi-objective), to develop two new B-MOBO variants. These new methods are then compared against their sequential counterparts and established B-MOBO methods using both computation time and real-time analysis. Results show that the new B-MOBO methods are as effective as existing methods at solving optimization problems in terms of solution quality and show improvements when real-time is considered.
Adelle Holder, H. Debruin, Jesse M. Sestito· PLoS ONE· 0 citations
