Gradient descent with biased rounding errors converges faster under certain conditions.
problem Stagnation or negative impact of rounding errors in neural network training with low precision.
method Analysis of gradient descent with stochastic fixed-point rounding errors under the Polyak-Lojasiewicz inequality.
result Biased rounding errors can improve convergence rates, especially when the Polyak-Lojasiewicz inequality holds.
AdaGrad-Norm achieves linear convergence for certain functions.
problem Proving linear convergence for specific types of functions.
method Introducing RUIG, a measure of gradient balance; developing a two-stage framework.
result AdaGrad-Norm achieves linear convergence for certain functions.
Study of non-convex potential functions in deep learning with Poincaré inequality.
problem Understanding convergence of stochastic dynamics in non-convex potential landscapes.
method Introduced log-Polyak-Lojasiewicz (log-PL) measures and analyzed their convergence properties.
result Langevin dynamics converges at a rate of O~(1/ε) for sufficiently small ε. AdamL optimizes deep learning models by incorporating loss function information.
problem Adaptive optimizers can suffer from poor generalization due to nonuniform gradient scaling.
method AdamL is a novel adaptive optimizer that considers loss function information for better generalization.
result AdamL achieves faster convergence or lower objective function values compared to other optimizers.
New analysis shows GMD can converge linearly under PL-like conditions.
problem Establishing linear convergence for generalized mirror descent.
method PL-based analysis for time-dependent mirrors, Taylor-series approach for stochastic GMD.
result Linear convergence of stochastic GMD under PL-like conditions.
New function class characterizes loss landscape of deep neural networks without over-parametrization.
problem Complex loss landscape of deep neural networks without over-parametrization.
method Proposed a novel class of functions to characterize loss landscape without over-parametrization.
result Gradient-based optimizers possess theoretical guarantees of convergence under the new function class assumption.
Polyak step size GD reaches final radius of convergence after log iterations.
problem Statistical and computational complexities of Polyak step size GD.
method Generalized smoothness and Lojasiewicz conditions, stability of gradients.
result Polyak step size GD reaches final statistical radius of convergence after logarithmic number of iterations.
Study on Wasserstein gradient flow for MMD between Coulomb measures.
problem Analyzing the long-time behavior of MMD between probability and target measures using Coulomb kernels.
method Existence of global weak solutions, ultracontractive estimate, regularity analysis, exponential decay proof, defective Polyak-Lojasiewicz inequality.
result Exponential decay of squared MMD toward a uniformly positive target measure on flat torus.
Harmonic map flow's singularity properties proven with Lojasiewicz inequalities.
problem Finite-time singularities of harmonic map flow in critical dimensions.
method Proving a weighted Lojasiewicz inequality.
result Continuity of body map and no-neck property for bubble-tree decompositions.
Quantized Stochastic Primal-Dual Methods for Distributed Optimization
problem Distributed optimization with stochastic gradients and finite-bit communication
method q-PDGD, a quantized stochastic primal-dual method
result Linear contraction to an explicit neighborhood under RSI, O(1/k) convergence under PL inequality
The Lojasiewicz inequalities for real analytic functions on Euclidean space were first proved by Stanislaw Lojasiewicz (1965) using methods of semianalytic and subanalytic sets, arguments later simplified by Bierstone and Milman (1988). In this article, we first give an elementary geometric, coordinate-based proof of t…
We prove several abstract versions of the Lojasiewicz-Simon gradient inequality for an analytic functional on a Banach space that generalize previous abstract versions of this inequality, weakening their hypotheses and, in particular, the well-known infinite-dimensional version of the gradient inequality due to Lojasie…
Study proves Lojasiewicz inequalities for harmonic maps near simple bubble trees.
problem Analyzing harmonic maps near simple bubble trees.
method Proves Lojasiewicz inequalities for harmonic maps close to simple bubble trees.
result Obtains new results on the convergence of harmonic map flow and energy spectrum.
We apply our abstract gradient inequalities developed by the authors in arXiv:1510.03817 to prove Lojasiewicz--Simon gradient inequalities for the harmonic map energy function using Sobolev spaces which impose minimal regularity requirements on maps between closed, Riemannian manifolds. Our Lojasiewicz--Simon gradient …
In this sequel to arXiv:1510.03817, we apply our abstract Lojasiewicz-Simon gradient inequality to prove Lojasiewicz-Simon gradient inequalities for coupled Yang-Mills energy functions using Sobolev spaces which impose minimal regularity requirements on pairs of connections and sections. The Lojasiewicz-Simon gradient …
This work ensures policy gradient methods converge to global optima for certain control problems.
problem Non-convex optimization challenges in policy gradient methods for complex control problems.
method Identifies structural properties ensuring non-convex objective functions have no suboptimal stationary points.
result Policy gradient methods converge to global optima under certain conditions, satisfying a Polyak-Lojasiewicz condition.
Stochastic GD converges linearly for CV@R learning under certain conditions.
problem Optimizing CV@R in statistical learning with non-convex loss functions.
method Stochastic Gradient Descent with Polyak-Łojasiewicz condition.
result Stochastic GD achieves linear convergence for CV@R learning.
Uniqueness of nondegenerate blowups for planar networks shown.
problem Uniqueness of nondegenerate blowups for the motion by curvature of planar networks.
method Proof based on Lojasiewicz-Simon gradient inequality applied to stability properties of critical points of the length functional.
result Uniqueness of nondegenerate compact blowups for the motion by curvature of planar networks.
Analyze SGD with biased gradients, improving convergence rates and accuracy.
problem Analyzing the convergence of SGD with biased gradients.
method Derive convergence results for smooth non-convex functions and quantify the impact of bias magnitude.
result Improved rates under the Polyak-Lojasiewicz condition and insights into how bias magnitude affects accuracy and convergence.
This paper explores geometric and analytic aspects of Lojasiewicz inequalities on vector bundles.
problem Understanding growth and stability conditions for real-analytic functions over vector bundles.
method Outline theory of functionals and variational problems over vector bundles, explore applications to real-analytic functionals.
result Describes the energy functional on $S^{n-1$ as a functional over a vector bundle.
The paper proves a Lojasiewicz inequality for maps from the 2-sphere to itself.
problem Analyzing maps from the 2-sphere to itself using Lojasiewicz inequalities.
method Using Lojasiewicz-Simon inequalities and Topping's repulsion estimates, along with a bubble-tree induction argument.
result Polynomial convergence of weak solutions of harmonic map flow on compact domains.
Gradient methods work well on overparameterized diagonal linear networks.
problem Understanding why gradient-based methods work well in overparameterized models.
method Study of Deep Diagonal Linear Networks with gradient flow analysis.
result Gradient flow on layer parameters induces a mirror-flow dynamic in the effective parameter space, leading to explicit convergence guarantees.
Improved greedy 2-coordinate updates for optimization problems with constraints.
problem Minimizing smooth functions subject to constraints.
method Exploiting a connection to steepest descent in the 1-norm, we give faster convergence rates and efficient computation.
result Greedy selection converges faster than random selection and can be computed in O(nlogn) time. It is a consequence of the Morse-Bott Lemma on Banach spaces that a smooth Morse-Bott function on an open neighborhood of a critical point in a Banach space obeys a Lojasiewicz gradient inequality with the optimal exponent one half. In this article we prove converses for analytic functions on Banach spaces: If the Loja…
SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.
problem Optimal convergence rates for non-convex functions under gradient noise.
method Geometric interpretation of the PL-condition to analyze convergence rates.
result Convergence rates of SGD and stochastic gradient descent match those of strongly convex quadratics.
SAIL-RevKL improves SAIL's convergence by regularizing the objective function.
problem Convergence of self-improving online LLM alignment algorithms.
method Proposed SAIL-RevKL, a regularized objective function to improve optimization landscape.
result Proved SAIL-RevKL satisfies the Polyak-Lojasiewicz (PL) condition with near-linear sample complexity.
Gradient descent converges linearly for overparameterized linear networks.
problem Convergence of gradient descent for overparameterized neural networks.
method Local Polyak-Lojasiewicz and Descent Lemma for overparameterized linear models.
result Gradient descent achieves linear convergence for two-layer linear networks under relaxed assumptions.
Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.
problem Local convergence of gradient descent in finite-width networks.
method Positive Neural Tangent Kernel (NTK), local Polyak-Łojasiewicz inequality, fixed-step containment in Locally Quasi-Convex Region (LQCR).
result Linear convergence achieved under specific conditions.
New method trains shallow neural networks with subquadratic width scaling.
problem Training shallow neural networks with optimal width scaling.
method Polyak-Lojasiewicz condition, smoothness, standard data assumptions, random matrix theory.
result Subquadratic scaling on network width with standard initialization strategies.
The elastic flow of curves converges smoothly to a critical point.
problem Smooth convergence of elastic flow of curves.
method Application of Lojasiewicz-Simon inequality.
result Smooth convergence to a critical point.
For any compact Lie group G and closed, smooth Riemannian manifold (X,g) of dimension d≥2, we extend a result due to Uhlenbeck (1985) that gives existence of a flat connection on a principal G-bundle over X supporting a connection with Lp-small curvature, when p>d/2, to the case of a connection with …
Study on stability of ALE Ricci-flat metrics using a modified Perelman's λ-functional.
problem Stability and instability of ALE Ricci-flat metrics.
method Use of a modified Perelman's λ-functional and Lojasiewicz inequality.
result Demonstrates dynamical instability of ALE Ricci-flat metrics.
Study on Riemannian isoperimetric inequality, finding it true generically.
problem Direct analogue of Euclidean isoperimetric inequality is false on Riemannian manifolds.
method Used Lojasiewicz-Simon inequality for real analytic metrics.
result Modified quantitative isoperimetric inequality holds for real analytic metrics.
Unified framework for analyzing neural networks trained by gradient descent.
problem Lack of generalizable guarantees for neural networks trained by gradient descent.
method Proxy convexity and proxy Polyak-Lojasiewicz inequalities.
result Unified guarantees for neural networks trained by gradient descent.
Improved SGD bounds for machine learning models with Markovian noise.
problem Uniform high-probability bounds for SGD under PL condition with Markovian noise.
method Combining Poisson equation for Markovian noise and probabilistic induction for almost-sure bounds.
result Matching 1/k decay rate for expected suboptimality. MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
New algorithms improve convergence of minimax optimization.
problem Minimax optimization convergence issues in nonconvex problems.
method Established new convergence results for two single-loop algorithms.
result Improved convergence rates for minimax optimization.
Paper proposes WD-DP ERM for distributed learning with improved privacy and performance.
problem Training models in distributed settings with privacy and performance guarantees.
method Weighted distributed differential privacy (WD-DP) for ERM, considering different weights of clients.
result Improved noise bound and excess empirical risk bound in distributed settings.
The Willmore flow preserves surface volume, leading to convergence to a sphere.
problem Long-term behavior of volume-preserving Willmore flow on surfaces.
method Volume-preserving Willmore flow, blow-up analysis, constrained Lojasiewicz-Simon inequality.
result Smooth solutions exist for spherical surfaces with Willmore energy below 8π and converge to a sphere.
Paper analyzes convergence rates of SGD for non-convex functions under various assumptions.
problem Analyzing convergence rates of SGD for non-convex functions.
method Studied convergence properties of Stochastic Gradient Descent (SGD) for invex functions under weaker and stronger hypotheses.
result Derives estimates on the rate of convergence of $J(oldsymbolθ_t)$ to its limit for functions satisfying the Polyak-Lojasiewicz (PL) condition.
In real algebraic geometry, Lojasiewicz's theorem asserts that any integral curve of the gradient flow of an analytic function that has an accumulation point has a unique limit. Lojasiewicz proved this result in the early 1960s as a consequence of his gradient inequality. Many problems in calculus of variations are que…
New functional proves mass positivity for ALE metrics.
problem Proving mass positivity for ALE metrics with Ricci-flat deformations.
method Introduced a new functional λALE and proved its monotonicity and Lojasiewicz-Simon inequality. result Established that small perturbations of Ricci-flat ALE metrics with nonnegative scalar curvature have nonnegative mass.
Develops a generalized version of Chung's Lemma for stochastic optimization methods.
problem Establishing asymptotic convergence rates for stochastic optimization methods under various step size rules.
method Generalized version of Chung's Lemma for a broader family of step size rules.
result Demonstrates tight non-asymptotic convergence rates for various stochastic methods.
Paper tackles fast convergence for non-convex strongly-concave min-max problems.
problem Non-convex strongly-concave min-max problems in deep learning.
method Proximal stage-based method with PL condition for faster convergence.
result Established fast convergence in primal objective gap and duality gap.
New method tackles nonconvex-nonconcave problems with local KL condition.
problem Nonconvex-nonconcave minimax problems under varying KL conditions.
method Inexact proximal gradient method for KL-structured subproblems.
result Complexity guarantees for approximate stationary points.
Predictability enables efficient parallelization of nonlinear models.
problem Understanding which nonlinear state space models can be efficiently parallelized.
method Established a relationship between system dynamics and optimization problem conditioning, quantified by the largest Lyapunov exponent.
result Predictable systems can be evaluated in O((logT)2) time, improving over conventional sequential approaches. Paper introduces input perturbation for privacy in machine learning models.
problem Protecting both training data and model parameters while maintaining privacy.
method Add noise to training data and train with perturbed data for differential privacy.
result Achieves (ε,δ)-differential privacy on the final model with privacy on original data.
Paper proves strong uniqueness of cylindrical tangent flows near singularity in Ricci flow.
problem Proving strong uniqueness of cylindrical tangent flows near singularity in Ricci flow.
method Established Lojasiewicz inequality for pointed W-entropy under cylindrical geometry assumption. result Strong uniqueness of cylindrical tangent flows at first singular time of Ricci flow proved.