The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
Abstract
We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank separation} in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is $O(\|\boldsymbol{\alpha}\|^2)$ and Jacobian leakage $O(\|\boldsymbol{\alpha}\|_\infty)$, giving a controlled departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting ($\Delta_A<0.1$) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control ($G_{\text{stack}}$) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
Comparison with the running mean highlights the stationary insertion-length distribution, its time-homogeneous update, stationary coefficient profile, and fixed effective memory, and its time-homogeneous update, stationary coefficient profile, and fixed effective memory.
Let $\pi(\mathrm{d} x)\propto e^{-U(x)}\,\mathrm{d} x$ on $\mathbb R^d$, where $0<m\leq L<\infty$, $mI_d\preceq\nabla^2U(x)\preceq LI_d$, and $\kappa=L/m$. It is known that, under warm-start assumptions, fixed-step Metropolis-adjusted Langevin algorithm (MALA) with properly tuned step size has mixing time of order $\kappa \sqrt{d}$ up to logarithmic factors. By contrast, when the condition number is bounded away from one, no single fixed step size yields a matching spectral-gap lower bound of order $(\kappa \sqrt{d})^{-1}$ uniformly over this target class. We show that MALA with a uniformly randomized step size admits a spectral-gap lower bound of this size. At each iteration, the randomized-step MALA considered here draws $h$ uniformly from $(0,H)$ and performs one ordinary MALA transition with step size $h$. We show that, when $H$ is of order $(L\sqrt{d})^{-1}$, the right spectral gap of randomized-step MALA admits a lower bound of order \[ \frac{1}{\kappa\sqrt{d}\,[1+\log(d+1)+\log\kappa]}. \] The main new ingredient in the proof is a Cheeger-type inequality for aggregating estimates of the one-step flow of MALA out of measurable sets at various step-size scales. It allows the scale used to control the flow to depend on the set and avoids the additional loss that would result from first summing the flows and then applying the standard Cheeger inequality. This work was developed with substantial assistance from ChatGPT, which suggested the uniformly randomized-step approach, developed the principal proof arguments, and generated the simulation and Lean 4 code. The human author checked and verified the mathematical content and take full responsibility for the results.
For stochastic gradient descent (SGD) with a constant stepsize $\alpha$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrt{\alpha}$ scaling and a Gaussian limit as $\alpha\downarrow 0$. We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize $\alpha$, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an $\alpha$-dependent metric. When the minimizer $x_\star$ has local flatness exponent $m\ge2$, meaning that $\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d$ as $x\to x_\star$, we obtain a contraction bound with factor $1-c\alpha^{m-1}$, where $c>0$ is a constant. This recovers the factor $1-c\alpha$ in the quadratic case $m=2$. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale $\alpha^{1/m}$ and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation $$ dY_t=-h_0(Y_t)\,dt+\Sigma^{1/2}\,dB_t , $$ where $h_0$ is the limiting drift at the minimizer and $\Sigma$ denotes the asymptotic covariance. This recovers the Gaussian limit when $m=2$ and gives generally non-Gaussian stationary limits in the flat case $m>2$. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.
We generalize a construction of Yves Meyer of sparse crystalline measures arising from powers of a Blaschke factor. Starting from a recursion $f_n=\theta^n f_0$ on the unit circle where $\theta$ is an inner function, we project the Fourier coefficient array $\widehat{f_n}(k)$ to the real line by placing its entries at the frequencies $k+\alpha n$. We identify the role of model spaces in this construction: in Meyer's one-factor Blaschke recursion, the requirement that the coefficient array $\widehat{f_n}(k)$ vanish whenever $kn<0$ is equivalent to $f_0\in K_{zb_\lambda}$, and for general inner functions the condition $f_0\in K_{z\theta}$ yields a purely atomic Radon measure with locally finite support and polynomial growth on the Fourier side. We also show that, when $f_0$ is holomorphic in an annulus containing the unit circle, exponential Fourier decay is sufficient to obtain a purely atomic Radon measure of polynomial growth, though not necessarily locally finite support. For finite Blaschke products, the coefficient recursion gives an explicit annihilating exponential polynomial whose zero set controls the support and separation of the inverse Fourier transform. This yields Meyer-Blaschke-type crystalline measures and Poisson identities with sampling and finite-truncation consequences.
We revisit Decomposition-Invariant Conditional Gradient methods, originally introduced by Garber and Meshi in 2016, for minimizing a convex and $\beta$-smooth function over a polytope in $\reals^n$, under an $\alpha$-quadratic growth condition. For 2-level polytopes we design a simple and parameter-free dyadic step-size rule that yields a linear convergence rate which scales with the dimension of the optimal face and not with the ambient dimension as in standard away-step-based conditional gradient methods for polytopes. For general polytopes, under a slightly stronger condition of $\alpha_{\mathrm{F}}$-\textit{facial quadratic growth}, we introduce a method whose number of iterations to reach an $\epsilon$-approximate solution is of the order $n+\frac{\beta{}D^2}{\alpha{}r^{*2}} + \frac{(d^*+1)\beta{}D^2}{\alpha_{\mathrm{F}}}\log(1/\epsilon)$, where $d^*$ is the dimension of the optimal face, $r^*$ is a separation parameter between the optimal set and faces that do not contain an optimal solution, and $D$ is the diameter of the polytope. This method is also parameter-free and only relies on standard line-search computations. The second result improves upon previous conditional gradient methods, whose number of iterations to $\epsilon$-approximation scales with $\frac{\beta{}D^2n}{\alpha}\log(1/\epsilon)$, in a meaningful regime $\max\{\frac{\alpha}{\alpha_{\rm F}}(d^*+1), \frac{1}{r^{*2}}\} \ll n$
For finite-horizon tabular CVaR reinforcement learning, prior work proves a $\widetilde{O}(\tau^{-1}\sqrt{SAK})$ leading regret bound for arbitrary normalized return laws and the sharper $\widetilde{O}(\sqrt{SAK/\tau})$ rate under a density lower bound. We show that the same Bernstein CVaR-UCBVI algorithm attains the sharper rate without continuity assumptions. The key is a selected-budget self-bound: the conditional variance of the episode shortfall is at most $\tau$ plus the value-estimation width. Substitution into the original Bernstein decomposition yields, with high probability, $\widetilde{O}(\sqrt{SAK/\tau}+(SAHK^{1/4}+S^2AH)/\tau)$ regret for arbitrary normalized return laws, including atomic, mixed, and continuous laws. The $\tau^{-1/2}$ leading term matches the expected-regret minimax lower bound up to logarithmic factors. Thus Bernstein CVaR-UCBVI is minimax-optimal over the full return-law class in the leading-order regime; the lower-order terms retain their $\tau^{-1}$ dependence.
Yuan-Long Chen· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
