An Approximate Conjugate Subgradient Algorithm with Matrix Parameter for Derivative-Free Nonsmooth Optimization Problems

Global convergence of a coderivative-based regularized Newton method with damping for nonsmooth optimization

A Modified Three-Term Hestenes–Stiefel Conjugate Gradient Method with Guaranteed Descent Property

Conjugate gradient (CG) methods are among the most widely used iterative algorithms for large-scale unconstrained optimization, valued for their simplicity and minimal memory requirements. A central challenge is ensuring guaranteed sufficient descent of the search direction, independent of the line search procedure. This paper proposes a Modified Three-Term Hestenes–Stiefel (MTTHS) conjugate gradient method that enforces the sufficient descent condition exactly, without any line search restriction. The proposed direction incorporates a geometric correction term θₖyₖ into the standard three-term framework, where the scalar θₖ is derived analytically to satisfy gₖ₊₁ᵀdₖ₊₁ = −‖gₖ₊₁‖². Global convergence is established under the strong Wolfe line search conditions via the Zoutendijk convergence criterion. Numerical experiments were based on 23 test problems, each solved at five different dimensions, n ∈ {100, 500, 1000, 5000, 10000}, yielding 115 test cases. The results show that the MTTHS method is robust and computationally efficient

A Local-Linearly Convergent Algorithm for Nonconvex Equality-Constrained Optimization

For solving nonconvex equality-constrained optimization problems, a recent Gradient-Eigenstep Algorithm by Goyens et al.~is an iteration-efficient approach, based on minimizing Fletcher's augmented Lagrangian function, for finding an approximate second-order stationary point from an arbitrary starting point. In this paper, the analysis of this algorithm is extended, offering a two-fold contribution. First, it is shown that a local-linear rate of convergence can be obtained by this method if it is initiated sufficiently close to a strong second-order stationary point and employs a sufficiently small step-size parameter and sufficiently large penalty parameter. In this case, the algorithm reduces to a gradient descent algorithm applied to minimize Fletcher's augmented Lagrangian. Second, as a particularly useful application of the first result, it is shown that the Gradient-Eigenstep algorithm can be used as an iteration-efficient subproblem solver in the context of a progressive sampling strategy for solving equality-constrained optimization problems when the objective and constraint functions are defined by large sample averages, ultimately offering an algorithm with an improved worst-case sample complexity when compared to an approach that solves a full-sample problem directly.

A Minimal-Gradient Subspace Method for Unconstrained Optimization

PANDA: A Matrix-Free Differentiable NMPC Solver via Proximal Averaged Quasi-Newton with Adaptive Linesearch Algorithm

Differentiable nonlinear model predictive control (NMPC) provides a principled way to embed optimal control structure into end-to-end learning paradigms, but its practical use is often limited by the computational and memory costs of both forward optimization and backward sensitivity propagation. This brief proposes PANDA, a matrix-free solver for differentiable NMPC. In the forward pass, PANDA combines proximal-gradient iterations with quasi-Newton acceleration and introduces an adaptive stepsize enlargement mechanism to mitigate the conservativeness of monotone stepsize reduction. The resulting stepsize behavior and its effect on local convergence are theoretically analyzed. In the backward pass, PANDA performs implicit differentiation from the residual equation and computes adjoint sensitivities using Krylov-subspace iterative methods together with automatic-differentiation-based Matrix-Vector product operators, thereby avoiding explicit Hessian and Jacobian construction. The method is evaluated on a nonconvex trailer NMPC problem embedded in an imitation learning task. The results show that PANDA achieves much faster forward and backward computation and lower memory overhead than representative differentiable optimization solvers, while maintaining effective imitation learning performance.

A Fixed-Penalty Linearized Augmented Lagrangian Method with Classical Multiplier Updates

Augmented Lagrangian methods are effective for nonlinear equality-constrained optimization, but solving their nonlinear primal subproblems can be expensive. For smooth nonconvex problems with deterministic or stochastic objectives, we propose a nonlinear-residual linearized augmented Lagrangian method (NR-LALM) that replaces this subproblem by a regularized Gauss-Newton-type step while retaining the classical multiplier update based on the nonlinear constraint residual. The resulting step is computed from one symmetric positive-definite linear system, but the mismatch between the linearized primal model and the nonlinear-residual update produces a quadratic constraint-linearization error in the multiplier identity. We show that this error can be controlled under local regularity; multiplier boundedness and trajectory localization are derived rather than assumed. With fixed, accuracy-independent parameters, deterministic NR-LALM finds an $\varepsilon$-approximate Karush-Kuhn-Tucker (KKT) pair in $O(\varepsilon^{-2})$ iterations and first-order oracle evaluations. For stochastic objectives, a projected stochastic path-integrated differential estimator with safeguarded restarts requires, in expectation, $O(\varepsilon^{-3})$ stochastic-gradient evaluations and $O(\varepsilon^{-2})$ constraint and Jacobian evaluations. Compactness and a Kurdyka-Lojasiewicz condition further yield finite-length convergence of the deterministic primal-dual sequence. An optional minimum-norm second-order correction reduces the constraint-linearization error from second to fourth order without changing the complexity orders. All theoretical results are formalized in Lean 4. Numerical experiments confirm the predicted error orders and show favorable performance on high-dimensional deterministic and stochastic problems.