Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates

The approach proposed in this paper enables a more capable DDDAS paradigm by improving the efficiency of the data-model-optimization loop by extracting a generalization bound based on Rademacher complexity that reveals the role of the $k-neighborhoods and related parameters.

Normalized First-Order Methods for Convex (L0, L1)-Smooth Optimization with Inexact Gradients

This work develops comparison-oracle variants of Normalized Gradient Descent and Gradient Descent with Polyak stepsizes and establishes explicit upper bounds on the approximation error that guarantee convergence and derive convergence rates for all proposed methods.

An Adaptive Projected-Gradient Algorithm for Sample-Average Approximations of Stochastic Multi-Objective Optimization

We consider stochastic multi-objective optimization over a nonempty closed convex set, where every objective is an expectation and only sample-gradient information is available. We develop a line-search-free and function-value-free adaptive projected-gradient algorithm for the sample-average approximation (SAA) problem. Each iteration computes a feasible regularized multi-gradient step and updates the regularization parameter from the projected step length. A normal-cone-based certificate yields descent estimates and an explicit complexity bound for the Pareto-stationarity residual of the SAA problem. The consistency of SAA gradients then transfers vanishing SAA residuals to Pareto stationarity for the population problem, while an additional concentration argument gives a finite-sample residual bound on compact sets. Experiments on synthetic problems, classification, portfolio selection, multi-task learning, and robot control illustrate the practical performance of our algorithm.

Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity

This work proposes a hypergradient-based bilevel RL algorithm using the optimality of the Boltzmann policy for the entropy regularized discounted RL objective function and obtains an iteration complexity of $O(\epsilon^{-1})$ and state-of-the-art sample complexity of $\tilde{O}(\epsilon^{-2})$ under mild regularity conditions.

High-Probability Convergence of SGD via Batched Updates

Stochastic gradient descent (SGD) is the primary workhorse for large-scale optimization. While the average behavior of its iterates, typically characterized by mean-squared error bounds, is well-understood, obtaining high-probability guarantees for the last iterate remains challenging. Prior approaches to this problem have either imposed restrictive assumptions (such as bounded domains or gradients) or relied on complex proofs involving auxiliary sequences. In this work, we propose Batched SGD, a simple variant that partitions online samples into epochs and performs a single update per epoch using a refined, low-variance gradient estimate. Our main contribution demonstrates that this batching mechanism enables a surprisingly simple high-probability analysis that avoids both restrictive assumptions and auxiliary sequences. Under standard smoothness and norm-sub-Gaussian noise assumptions, we establish near-optimal rates for both strongly convex and non-convex objectives. Furthermore, we show that our batching idea extends naturally to federated learning (FL). We provide the first high-probability guarantees for FL, achieving logarithmic communication complexity, linear speedup in the number of agents, and resilience to data heterogeneity.

Hyper-Gradient Methods for Bilevel Optimization with Manifold Lower-level Solution Set