arXiv research

HA-SME models SGD dynamics with Hessian info for better escaping behaviors.

Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this …

The roundworm C. elegans exhibits robust escape behavior in response to rapidly rising temperature. The behavior lasts for a few seconds, shows history dependence, involves both sensory and motor systems, and is too complicated to model mechanistically using currently available knowledge. Instead we model the process p…

Study of SGD with state-dependent noise, improving escape from local minima.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the precondition…

This research explains why SGD generalizes better than ADAM in deep learning.

Improves training GANs by escaping limit cycles.

Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point tha…

SGD's escape rate depends on log loss barrier, not linear loss barrier.

Classifies conformal transformations in spacetimes without observer horizons.

The paper analyzes neural network dynamics after weights escape the origin.

We shortly review the statistical properties of the escape times, or hitting times, for stock price returns by using different models which describe the stock market evolution. We compare the probability function (PF) of these escape times with that obtained from real market data. Afterwards we analyze in detail the ef…

Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.

Gradient descent can take exponentially long to escape saddle points in 2D.

New algorithm helps escape saddle points in optimization problems.

The paper proves Zimmer's conjecture for non-uniform lattices by controlling mass escape and Lyapunov exponents.

New methods help escape strict saddle points in nonsmooth optimization.

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

Geodesic loops escape from balls at a sublinear rate imply virtually abelian fundamental group.

Houdini finds high-dimensional saddle points under few constraints.

The Dirichlet random walk on manifolds has a positive escape rate if the cover is non-amenable.

New result on group actions in CAT(0) spaces with vanishing escape rate.

A new algorithm trains deep neural networks by adding neurons greedily.

We study some potential theoretic properties of homothetic solitons $Σ^n$ of the MCF and the IMCF. Using the analysis of the extrinsic distance function defined on these submanifolds in $\mathbb{R}^{n+m}$, we observe similarities and differences in the geometry of solitons in both flows. In particular, we show that par…

We solve the escape problem for the Heston random diffusion model. We obtain exact expressions for the survival probability (which ammounts to solving the complete escape problem) as well as for the mean exit time. We also average the volatility in order to work out the problem for the return alone regardless volatilit…

New analysis of SGD with MCMC gradient estimator shows convergence rate and saddle point escape.

We study the mean escape time in a market model with stochastic volatility. The process followed by the volatility is the Cox Ingersoll and Ross process which is widely used to model stock price fluctuations. The market model can be considered as a generalization of the Heston model, where the geometric Brownian motion…

This paper proposes a new global optimization algorithm using deep learning.

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…

Hill-ADAM optimizes loss landscapes by exploring state space deterministically.

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…

Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…

A new method helps escape saddle points in non-convex optimization.

The paper analyzes how noise geometry influences the performance of SGD in machine learning.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…

New metrics help predict Brownian motion on surfaces and higher dimensions.

We study the detailed path-wise behavior of the discrete-time Langevin algorithm for non-convex Empirical Risk Minimization (ERM) through the lens of metastability, adopting some techniques from Berglund and Gentz (2003. For a particular local optimum of the empirical risk, with an arbitrary initialization, we show tha…

A statistical analysis of financial, economic, and demographic indicators performed by the authors demonstrates (1) that the main countries of East Africa (Uganda, Kenya, and Tanzania) have not escaped the Malthusian Trap yet; (2) that this countries are not likely to follow the "North African path" and to achieve this…

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Two-layer networks learn hard GLMs with SGD in high dimensions.

New $k$-step policy gradient method avoids local optima in restricted policy classes.

Theoretical analysis explains why models generalize after overfitting in modular addition.

The expectation-maximization (EM) algorithm has been widely used in minimizing the negative log likelihood (also known as cross entropy) of mixture models. However, little is understood about the goodness of the fixed points it converges to. In this paper, we study the regions where one component is missing in two-comp…

Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining re…