arXiv research

Automated model predicts C. elegans escape behavior.

HA-SME models SGD dynamics with Hessian info for better escaping behaviors.

Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this …

Study of SGD with state-dependent noise, improving escape from local minima.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

Adaptive methods like Adam help escape saddle points in deep learning.

This research explains why SGD generalizes better than ADAM in deep learning.

Improves training GANs by escaping limit cycles.

Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point tha…

SGD's escape rate depends on log loss barrier, not linear loss barrier.

Classifies conformal transformations in spacetimes without observer horizons.

The paper analyzes neural network dynamics after weights escape the origin.

We shortly review the statistical properties of the escape times, or hitting times, for stock price returns by using different models which describe the stock market evolution. We compare the probability function (PF) of these escape times with that obtained from real market data. Afterwards we analyze in detail the ef…

Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.

Gradient descent can take exponentially long to escape saddle points in 2D.

New algorithm helps escape saddle points in optimization problems.

The paper proves Zimmer's conjecture for non-uniform lattices by controlling mass escape and Lyapunov exponents.

New methods help escape strict saddle points in nonsmooth optimization.

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

Geodesic loops escape from balls at a sublinear rate imply virtually abelian fundamental group.

Houdini finds high-dimensional saddle points under few constraints.

The Dirichlet random walk on manifolds has a positive escape rate if the cover is non-amenable.

New result on group actions in CAT(0) spaces with vanishing escape rate.

A new algorithm trains deep neural networks by adding neurons greedily.

We study some potential theoretic properties of homothetic solitons $Σ^n$ of the MCF and the IMCF. Using the analysis of the extrinsic distance function defined on these submanifolds in $\mathbb{R}^{n+m}$, we observe similarities and differences in the geometry of solitons in both flows. In particular, we show that par…

We solve the escape problem for the Heston random diffusion model. We obtain exact expressions for the survival probability (which ammounts to solving the complete escape problem) as well as for the mean exit time. We also average the volatility in order to work out the problem for the return alone regardless volatilit…

New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.

We study the mean escape time in a market model with stochastic volatility. The process followed by the volatility is the Cox Ingersoll and Ross process which is widely used to model stock price fluctuations. The market model can be considered as a generalization of the Heston model, where the geometric Brownian motion…

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…

This paper proposes a new global optimization algorithm using deep learning.

Hill-ADAM optimizes loss landscapes by exploring state space deterministically.

Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…

A new method helps escape saddle points in non-convex optimization.

The paper analyzes how noise geometry influences the performance of SGD in machine learning.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Heavy-ball algorithms can always avoid saddle points with random initialization.

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

New metrics help predict Brownian motion on surfaces and higher dimensions.

This paper compares EM and GD in two-component mixture models, finding EM escapes bad local optima more reliably.

We study the detailed path-wise behavior of the discrete-time Langevin algorithm for non-convex Empirical Risk Minimization (ERM) through the lens of metastability, adopting some techniques from Berglund and Gentz (2003. For a particular local optimum of the empirical risk, with an arbitrary initialization, we show tha…

Riemannian gradient descent helps escape saddle points on curved spaces.

A statistical analysis of financial, economic, and demographic indicators performed by the authors demonstrates (1) that the main countries of East Africa (Uganda, Kenya, and Tanzania) have not escaped the Malthusian Trap yet; (2) that this countries are not likely to follow the "North African path" and to achieve this…

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Two-layer networks learn hard GLMs with SGD in high dimensions.

NADPEx uses dropout to enable temporally consistent exploration in reinforcement learning.

Theoretical analysis explains why models generalize after overfitting in modular addition.

New $k$-step policy gradient method avoids local optima in restricted policy classes.