This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…
We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…
SGD converges to global minimum for certain non-convex functions.
problem Theoretical challenges in optimizing non-convex functions in machine learning.
method Perturbed SGD on a broad class of non-convex functions.
result SGD converges to global minimum for certain non-convex functions.
Difference of convex (DC) functions cover a broad family of non-convex and possibly non-smooth and non-differentiable functions, and have wide applications in machine learning and statistics. Although deterministic algorithms for DC functions have been extensively studied, stochastic optimization that is more suitable …
Lower bounds for higher-order methods in non-convex optimization.
problem Proving lower bounds for higher-order methods in smooth non-convex finite-sum optimization.
method Analyzing deterministic and randomized algorithms, proposing a new smoothness assumption.
result Proves optimal lower bounds for simulating pth-order regularized methods on the whole function.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
Improved convergence analysis for decentralized non-convex optimization.
problem Minimizing a sum of smooth non-convex functions over a network.
method Gradient tracking in decentralized stochastic gradient descent (GT-DSGD).
result GT-DSGD achieves network-independent performances matching centralized SGD under certain conditions.
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. In this two-part work, we propose an algorithmic framework for solving non-convex problems whose objective function is the sum of a number of smooth component functions plus a convex (possibly non-smooth) or/and smooth (possibly non-convex) regularization function. The proposed algorithm incorporates ideas from several…
Study revisits AdaGrad convergence with relaxed noise assumptions.
problem Non-convex smooth optimization problems with general noise.
method General noise model with function value gap and gradient magnitude control.
result Probabilistic convergence rate of ( ilde{\mathcal{O}}(1/\sqrt{T})) under general noise.
This paper analyzes saddle points and minimax points in non-convex smooth games.
problem Understanding local optimal points in non-convex smooth games.
method Comprehensive analysis of local minimax points, including their optimality conditions and stability.
result Local saddle points are uniformly local minimax points under mild continuity assumptions.
Online SGD from random init solves non-smooth, non-convex phase retrieval.
problem Solving phase retrieval with non-smooth, non-convex loss functions.
method Online stochastic gradient descent (SGD) with constant step size, starting from arbitrary initialization.
result SGD converges from arbitrary initializations for the amplitude squared loss objective.
New DRO algorithm finds robust models without strong assumptions.
problem Learning robust models against distribution shift.
method Non-convex, non-smooth, non-asymptotic convergence guarantees for DRO.
result Algorithm finds first-order stationary points efficiently.
New method for faster convergence in non-convex optimization with unbounded smoothness.
problem Finding first-order stationary points of non-convex functions with unbounded smoothness.
method Developed a stopped analysis technique to prove convergence rates for ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth functions. result Achieved O ( p o l y log ( T ) T ) \mathcal{O}(\frac{\mathrm{poly}\log(T)}{\sqrt{T}}) O ( T poly l o g ( T ) ) convergence rates without uniform noise bounds. We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of conver…
Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…
Improved optimization technique reduces training complexity for non-convex problems.
problem Training non-convex optimization problems with exploding gradients.
method Employed variance reduction technique (SPIDER) with carefully designed learning rate.
result Improved stochastic gradient complexity to O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) for ε ε ε -stationary solutions. Lower bounds on queries needed for finding stationary points in non-convex optimization.
problem Finding ε ε ε -stationary points in non-convex stochastic optimization. method Proving lower bounds on the number of queries required by stochastic first-order methods.
result Lower bounds on the number of queries required to find ε ε ε -stationary points are tight and optimal. Advances smooth over-parameterization for solving non-smooth optimization problems.
problem Non-smooth optimization with structural constraints in imaging and machine learning.
method Smooth over-parameterization of non-smooth problems, using gradient descent and mirror descent.
result Gradient descent on the reformulated smooth problem converges efficiently without parameter tuning.
An Euler discretization of the Langevin diffusion is known to converge to the global minimizers of certain convex and non-convex optimization problems. We show that this property holds for any suitably smooth diffusion and that different diffusions are suitable for optimizing different classes of convex and non-convex …
New sampling algorithm for non-smooth potentials.
problem Sampling from non-smooth potentials.
method Proximal algorithm based on rejection sampling.
result Achieves better complexity than existing methods.
In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…
In this paper, we study a family of non-convex and possibly non-smooth inf-projection minimization problems, where the target objective function is equal to minimization of a joint function over another variable. This problem include difference of convex (DC) functions and a family of bi-convex functions as special cas…
Paper improves stability analysis of SGD for various loss functions and data distributions.
problem Improving stability analysis of SGD for non-convex loss functions and data distributions.
method Analyzes stability of SGD for convex and non-convex loss functions, and improves data-dependent bounds.
result Improved stability bounds for non-convex loss functions and convex regularized loss functions.
Finding efficient and provable methods to solve non-convex optimization problems is an outstanding challenge in machine learning and optimization theory. A popular approach used to tackle non-convex problems is to use convex relaxation techniques to find a convex surrogate for the problem. Unfortunately, convex relaxat…
New algorithm reduces complexity for optimizing complex machine learning tasks.
problem Optimizing complex machine learning objectives like reinforcement learning and portfolio management.
method Developed SARAH-Compositional algorithm using Stochastic Recursive Gradient Descent.
result Achieved optimal IFO complexity bounds for stochastic compositional optimization.
Optimizes deep learning pipelines with novel algorithms for smooth and non-smooth functions.
problem Optimizing deep learning pipelines for smooth and non-smooth functions.
method Provided matching lower and upper bounds for smooth convex and non-convex functions, and developed PPRS for non-smooth convex functions.
result PPRS achieves near-linear speed-up and convergence time for non-smooth non-convex problems.
SGD's uncertainty quantified in non-convex learning problems.
problem Uncertainty quantification in non-convex learning problems.
method Asymptotic normality of SGD iterates and bias characterization.
result SGD iterates are asymptotically normally distributed around the expected value of the invariant distribution.
New bounds found for optimizing non-convex functions with noisy data.
problem Limits of first-order stochastic optimization in non-convex settings.
method Divergence decomposition to construct challenging subclasses.
result Sharp lower bounds on noisy gradient queries for various non-convex classes.
This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.
problem Analyzing convergence of RMSProp and Adam in non-convex optimization with relaxed assumptions.
method Developed new convergence analyses for RMSProp and Adam, considering adaptive learning rates and affine noise variance.
result RMSProp and Adam converge to ε-stationary points with iteration complexities of O(ε^(-4)) under proper hyperparameters.
This paper studies quasar-convex functions to improve optimization methods.
problem Improving optimization methods for non-convex functions.
method Study of first order methods for quasar-convex functions.
result Proves complexity upper bounds similar to convex functions.
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
This paper analyzes SGD with biased gradients for deep learning models.
problem Analyzing SGD with biased gradients for deep learning models.
method Non-asymptotic analysis of SGD with adaptive steps for non-convex smooth functions.
result Adagrad, RMSProp, and AMSGRAD converge to critical points at a similar rate to unbiased case.
We introduce a new local regret framework for non-convex models in dynamic environments.
problem Challenges in online forecasting for non-convex models with frequent updates and concept drift.
method We propose a novel local regret framework and a time-smoothed gradient update rule.
result Our approach yields more stable, robust, and computationally efficient forecasting compared to state-of-the-art methods.
The paper explores various stationarity concepts in non-smooth optimization.
problem Understanding stationarity in non-smooth optimization problems.
method Introduction and discussion of different stationarity concepts for non-convex non-smooth functions.
result Clarification of the relationship among different stationarity concepts and their relevance in iterative methods.
Two algorithms find optimal points in decentralized optimization.
problem Decentralized non-convex stochastic optimization with composite objective functions.
method Prox-DASA and Prox-DASA-GT algorithms for finding ε-stationary points.
result Achieves comparable complexity without large batch sizes or complex per-iteration operations.
New algorithm solves non-convex, non-differentiable min-max games.
problem Limited theoretical understanding of non-smooth min-max games.
method Proximal gradient descent-ascent algorithm for convex-strongly convex games.
result Algorithm converges to ε-Nash equilibrium with polynomial gradient evaluations.
We design a stochastic algorithm to train any smooth neural network to ε \varepsilon ε -approximate local minima, using O ( ε − 3.25 ) O(\varepsilon^{-3.25}) O ( ε − 3.25 ) backpropagations. The best result was essentially O ( ε − 4 ) O(\varepsilon^{-4}) O ( ε − 4 ) by SGD. More broadly, it finds ε \varepsilon ε -approximate local minima of any smooth nonconvex function in …
Paper tackles non-convex constrained DRO with a stochastic algorithm for large-scale applications.
problem Training robust models against data distribution shifts with non-convex loss functions.
method Developed a stochastic algorithm for non-convex constrained DRO with a complexity independent of dataset size.
result Algorithm finds ε-stationary points with computational complexity of O(ε^(-3k_*-5)) for general Cressie-Read divergence.
Heavy Ball method speeds up finding global optima in non-convex problems.
problem Finding global optima in non-convex optimization problems.
method Heavy Ball momentum in non-convex optimization.
result Heavy Ball helps iterates enter a benign region faster, containing a global optimal point.
Given a nonconvex function that is an average of n n n smooth functions, we design stochastic first-order methods to find its approximate stationary points. The convergence of our new methods depends on the smallest (negative) eigenvalue − σ -σ − σ of the Hessian, a parameter that describes how nonconvex the function is. Our m…
New framework for robust hypothesis testing using Sinkhorn uncertainty sets.
problem Non-convex robust hypothesis testing problem.
method Exact mixed-integer exponential conic reformulation and convex approximation.
result Satisfactory testing performance and computational efficiency.
A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.
problem Sparse-penalized quantile regression with non-convex penalties.
method Single-loop smoothing ADMM (SIAD) algorithm for faster convergence.
result SIAD method outperforms existing approaches in solving sparse-penalized quantile regression.
New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.
problem Challenges in high-dimensional linear regression under heavy-tailed noise or outliers.
method Projected sub-gradient descent algorithm for sparse and low-rank regression problems.
result Algorithm achieves linear convergence and statistical optimality under various noise conditions.
New convergence analysis for ADAM algorithm in non-convex optimization with adaptive step size.
problem Convergence issues in ADAM algorithm for non-convex optimization.
method Study of ADAM algorithm under bounded adaptive step size assumption, providing safe step sizes.
result Novel first order convergence rate result in deterministic and stochastic contexts.
New SGD covering technique yields dimension-independent generalization bounds.
problem Generalization of stochastic gradient descent in non-convex, non-smooth settings.
method Localized ε-covers for SGD trajectories, showing dimension-independent complexity.
result Generalization error upper bounded by O ( ( log n log ( n P ) ) / n ) O(\sqrt{(\log n\log(nP))/n}) O ( ( log n log ( n P )) / n ) .