Damped Gauss Newton Search for Multi Metric Hyperparameter Optimization

Abstract

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a black-box, target-seeking damped Gauss--Newton method for multiple validation responses. The method estimates trajectory-based numerical sensitivities from successive changes in hyperparameters and validation performance. These full-vector observations construct an iterative secant approximation of the local sensitivity, and a damped Gauss--Newton system jointly updates all hyperparameters. The method does not differentiate through model training or require coordinate-wise perturbation runs: after two initial evaluations establish the first secant, each subsequent iteration requires one new full-vector evaluation. Tikhonov regularization stabilizes the underdetermined local inverse problem. Unlike multi-objective HPO that approximates a Pareto front, our formulation seeks a specified operating point in multi-response performance space. We evaluate the method on four-dimensional XGBoost HPO across three classification datasets, six-dimensional LoRA/SFT post-training of Qwen3-VL-8B on a binary VQAv2 subset, and eight-dimensional threshold optimization on fixed classifier outputs. XGBoost results are comparable to grid, random, and TPE search with fewer model evaluations. The VLM study reveals non-monotonic response and sensitivity trajectories, while the threshold study demonstrates target seeking in an underdetermined system and sensitivity to initialization. Overall, the results support trajectory-based secant sensitivity as an evaluation-efficient local optimization mechanism complementary to global black-box HPO.