An algorithm for composite optimization problems of the form min x f (x) + g(T x), where f is smooth and g may be non-smooth is proposed, which leverages the Moreau envelope to smooth the non-smooth component while adapting to problem geometry through linear minimization oracles.
Abstract
We propose MELMO (Moreau Envelope Smoothing with Linear Minimization Oracles), an algorithm for composite optimization problems of the form min x f (x) + g(T x), where f is smooth and g may be non-smooth. The method leverages the Moreau envelope to smooth the non-smooth component while adapting to problem geometry through linear minimization oracles. Assuming g is $\rho$-weakly convex, we establish a family of convergence bounds parameterized by the step-size and smoothing schedules, thereby making explicit the trade-off between optimizing the smoothed objective and recovering stationarity for the original composite problem. In particular, one regime yields O(k -1/4 ) rates for both the smoothed-gradient norm and a composite stationarity proxy, while another yields O(k -1/3 ) for the smoothed-gradient norm together with O(k -1/4 ) for the composite proxy. We also establish a K-horizon-dependent convergence rate that yields O(K -1/3 ) for the composite proxy. Empirically, MELMO is competitive with variable smoothing and subgradient baselines on sparse low-rank matrix factorization and image denoising.
Minimizing gradients of a convex function is an important problem across optimization and learning tasks. The gradient provides a directly computable certificate of approximate stationarity, and its minimization usually implies stronger results than those for minimization of function values. In this work, we study gradient-norm minimization for convex functions that are $(L,\kappa)$-H\"older smooth with respect to the $\ell_p$-norms, $p \geq 1$. We develop algorithms that achieve near-optimal gradient-oracle complexity for this problem. In the smooth case, our results resolve the previously open setting $p>2$. For H\"older-smooth objectives, we close the complexity gap throughout the full $p$-range, including to the best of our knowledge, a gap in the Euclidean case. We provide two families of algorithms: the first one comes with a simple iteration and generalizes a phenomenon known as mirror duality, exploiting dual behaviours of algorithms with errors and inexact computations. The second makes use of accumulating regularizers centered at different approximate solutions, which we sequentially minimize in order to provide our near-optimal rates.
Nico Pelleriti, Maryam Shiran, David Martínez-Rubio et al.· 0 citations
We study a class of weakly convex optimization problems in which the objective is the sum of a smooth convex term and a weakly convex term that may be nonsmooth. To exploit this structure, we develop a splitting technique based on the alternating direction method of multipliers (ADMM), which decouples the minimization of the two components into tractable subproblems. Because the update associated with the smooth term may require an inner iterative solver, we further linearize this term, yielding a linearized ADMM (LADMM) scheme with an inexpensive one-step update. Under mild conditions, we establish the subsequence convergence of both ADMM and LADMM methods to directional stationary solutions, which are equivalent to critical points and Clarke stationary solutions for our weakly convex problem. Numerical experiments on two low-dimensional test functions and a high-dimensional logarithmic regularized logistic regression model demonstrate that the proposed approaches are computationally efficient and produce solutions of comparable quality to baseline methods.
Sheng-Han Mei, Cheng-Yu Ke, Yifei Lou et al.· 0 citations
A global iteration complexity bound is established for the smallest gradient norm among the first $k$ iterates for the smallest gradient norm among the first $k$ iterates when the initial sublevel set is bounded.
We study convex-concave minimax optimization problems with bilinear coupling of the form $\min_{x\in \mathcal X}\max_{y\in \mathcal Y} \; f(x)+\langle y,\mathbf{B}x\rangle-g(y),$ where the functions $f$ and $g$ have H\"older continuous (sub)gradients. This setting covers a broad range of regimes, from nonsmooth problems with bounded subgradient variation to smooth problems with Lipschitz continuous gradients; for a smooth component used in the coupling-induced regularizer, its Lipschitz-gradient constant is assumed to hold in the ambient space. We propose a sliding method that exploits the composite structure of the problem by querying the oracles associated with $f$, $g$, and the bilinear coupling operator at prescribed frequencies determined by their individual properties. The method is based on a recursive sliding scheme for monotone variational inequalities. We establish convergence guarantees under H\"older continuity and show how the resulting complexity bounds depend explicitly on the H\"older exponents, H\"older constants, strong convexity parameters, and spectral properties of the coupling matrix. Our analysis covers nonstrongly convex and partially strongly convex regimes. For stochastic problems, we prove a uniform expected-gap bound in the degenerate regime and, under ambient smoothness and positive effective curvature, convergence up to an explicit noise floor. Numerical experiments reproduce the predicted H\"older exponents and confirm that the number of gradient evaluations required for each function separates according to its own smoothness level rather than the worse of the two. A tomographic benchmark shows runtime gains when gradient evaluations are more expensive than the additional matrix-vector products.
This work proposes a novel single-loop algorithm based on a constrained reformulation in which lower-level stationarity is imposed as a constraint, and constructs a regularized Lagrangian by introducing a quadratic regularizer and restricting the dual variable to a bounded domain.
This paper introduces four groups of subspace methods for nonlinear monotone equations, with applications to large-scale machine learning problems. The methods use Jacobian-free subspace ({\tt JFS}) directions of conjugate-gradient type, combined with either fixed step sizes or variable step sizes generated by the projected method of Solodov and Svaiter. To ensure convergence independently of the specific algebraic form of the subspace directions, we impose an angle condition together with an explicit scaling rule controlling the effective search directions. Under monotonicity and Lipschitz continuity of the operator, we establish global convergence for both the line-search and fixed-step frameworks, as well as a best-iterate residual rate $O(\ell^{-1/2})$. Under a local error bound, the distance to the solution set satisfies the sharper decay $o(\ell^{-1/2})$. If the operator is continuously differentiable and its Jacobian is nonsingular at a solution, the required local error bound and local isolation follow, yielding $R$-linear local convergence. The residual sequence then converges geometrically and hence satisfies the last-iterate rate $o(\ell^{-1})$, without strong monotonicity. We also derive iteration and residual-evaluation complexity bounds: $O(\varepsilon^{-2})$ for the baseline best-iterate guarantee and $O(\log(\varepsilon^{-1}))$ in the local linear regime, together with a uniform bound on backtracking residual evaluations. Under additional asymptotic assumptions, the proposed {\tt JFS} directions and several classical update directions admit related optimistic gradient descent--ascent (\texttt{OGDA})-type residual--memory representations. Numerical experiments illustrate the robustness and efficiency of the methods on representative min--max problems.
M. Kimiaei, Shima Shabani, Michael Breuß· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
