arXiv research

Gradient descent can take exponentially long to escape saddle points in 2D.

New methods help escape strict saddle points in nonsmooth optimization.

Houdini finds high-dimensional saddle points under few constraints.

Heavy-ball algorithms can always avoid saddle points with random initialization.

A new method helps escape saddle points in non-convex optimization.

Adaptive methods like Adam help escape saddle points in deep learning.

Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…

New algorithm helps escape saddle points in optimization problems.

The paper analyzes neural network dynamics after weights escape the origin.

PWGF escapes saddle points in nonconvex optimization.

Riemannian gradient descent helps escape saddle points on curved spaces.

Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…

Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points su…

New guarantees for SGD in non-convex optimization without strict noise bounds.

Simple gradient descent algorithm escapes saddle points efficiently.

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point tha…

DEO uses gradient information to escape saddle points in neural networks.

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

A new algorithm trains deep neural networks by adding neurons greedily.

This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i.e., it is almost "dimension-free"). The convergence rate of this procedure matches the well-known convergence rate of gradient descent to…

New algorithm finds local minima in non-convex, non-smooth problems.

In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…

Linear speedup achieved in non-convex optimization for decentralized systems.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the es…

New algorithms improve sampling from Bayesian deep learning models.

New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.

Paper shows faster convergence to local-minimizers in over-parametrized models under interpolation-like conditions.

This paper extends Newton's method to distributed learning, avoiding saddle points and handling Byzantine workers.

In this paper, we study the problem of escaping from saddle points in smooth nonconvex optimization problems subject to a convex set $\mathcal{C}$. We propose a generic framework that yields convergence to a second-order stationary point of the problem, if the convex set $\mathcal{C}$ is simple for a quadratic objectiv…

While first-order optimization methods such as stochastic gradient descent (SGD) are popular in machine learning (ML), they come with well-known deficiencies, including relatively-slow convergence, sensitivity to the settings of hyper-parameters such as learning rate, stagnation at high training errors, and difficulty …

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…

We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that - contrary to the case of isotropic noise - this variance is proportional to t…

Improved method finds second-order stationary points privately with better efficiency.

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these l…

The paper studies quadratic neural networks, proving existence of spurious minima and saddle points.

WSFN overcomes saddle points for non-convex functionals in Wasserstein space.

This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only $\mathcal{\tilde{O}}(ε^{-3.5})$ stochastic gradien…

Paper proposes faster method to find local minima in nonconvex optimization.

Online algorithms for identifying river pollution sources.

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

Rapid mixing of Langevin dynamics on Riemannian manifolds

SSRGD finds local minima in nonconvex problems with simple gradient updates.

A new method for growing neural networks by splitting neurons, improving efficiency.

Paper proposes estimating gradients for zeroth-order nonconvex optimization.