Leveraging High-Performance Scientific Computing for the Limited-Memory Hyperparameter Tuning of Large-Scale Unconstrained Continuous Optimization Solvers
TL;DR
This work investigates a data-driven selection of the L-BFGS memory parameter (mem) for large-scale unconstrained optimization and proposes a two-stage hybrid machine-learning framework that predicts intermediate solver quantities and then forecasts runtime for each candidate memory value.
Abstract
Optimization algorithms rely on hyperparameters that strongly influence convergence and computational performance. In practice, these values are often selected using fixed defaults or heuristic methods, potentially producing suboptimal results. This work investigates a data-driven selection of the L-BFGS memory parameter (mem) for large-scale unconstrained optimization. To generate our empirical dataset, we deployed a distributed high-performance computing workflow to evaluate 215 problem configurations across a discrete memory grid (mem ∈ {1, 2, . . . , 100}). This process systematically captured structural, profiling, and solver-performance features from a total of 21,500 complete optimization runs. We propose a two-stage hybrid machine-learning framework that predicts intermediate solver quantities and then forecasts runtime for each candidate memory value. On held-out test configurations, the resulting recommendations achieve median parity with the per-instance empirical oracle over the tested memory grid (Q(0.50) = 1.0000). Furthermore, our ensemble architecture systematically outperforms the standard package default (mem = 5), reducing the90th-percentile performance ratio to 2.0613, compared with 2.6839 for the default. These results suggest that instance-specific memory recommendation can improve over the default setting while approaching a retrospective global static reference selected with dataset-level hindsight.