Asymptotic Analysis of Gradient Mapping-type Stationarity Measure for the Sum of Nonconvex Nonsmooth Functions and Applications to Proximal Gradient-type Algorithms
TL;DR
It is shown that Fr\'echet or proximal stationarity can be characterized through asymptotic vanishing of the proposed measure along some convergent sequence and analogous asymptotic results for a smoothing-based variant of the measure are established.
Abstract
We propose a gradient mapping-type stationarity measure for the sum of two possibly nonsmooth nonconvex functions. Under suitable regularity assumptions, we show that Fr\'echet or proximal stationarity can be characterized through asymptotic vanishing of the proposed measure along some convergent sequence. We also establish analogous asymptotic results for a smoothing-based variant of the measure, which enables us to combine the proposed framework with smoothing techniques developed for nonsmooth optimization. As an application of our analysis of the stationarity measure, we provide an affirmative answer to an open question raised by [Olikier-Waldspurger, SIAM J. Optim., 2025] on whether every cluster point of a sequence generated by a proximal gradient method is a proximal stationary point under local Lipschitz smoothness of one component of the cost function. As a second application, we propose a proximal variable smoothing algorithm with a nonmonotone linesearch for minimizing the sum of two nonsmooth nonconvex functions under lower regularity of one component and prox-regularity of the other. For the proposed algorithm, we show that every cluster point of a subsequence such that the stationarity measure vanishes is a Fr\'echet stationary point.