A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Gradient descent can take exponentially long to escape saddle points in 2D.
Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this …
New methods help escape strict saddle points in nonsmooth optimization.
Study of SGD with state-dependent noise, improving escape from local minima.
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…
Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points su…
A new method helps escape saddle points in non-convex optimization.
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the precondition…
Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.
New method escapes local optima in neural architecture optimization.
SGD's escape rate depends on log loss barrier, not linear loss barrier.
Recent years have seen increased interest in performance guarantees of gradient descent algorithms for non-convex optimization. A number of works have uncovered that gradient noise plays a critical role in the ability of gradient descent recursions to efficiently escape saddle-points and reach second-order stationary p…
PWGF escapes saddle points in nonconvex optimization.
This paper improves neural network learning by escaping the NTK regime and efficiently learning sparse polynomials.
DEO uses gradient information to escape saddle points in neural networks.
Classifies conformal transformations in spacetimes without observer horizons.
The paper analyzes neural network dynamics after weights escape the origin.
We shortly review the statistical properties of the escape times, or hitting times, for stock price returns by using different models which describe the stock market evolution. We compare the probability function (PF) of these escape times with that obtained from real market data. Afterwards we analyze in detail the ef…
Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.
New algorithm helps escape saddle points in optimization problems.
The paper proves Zimmer's conjecture for non-uniform lattices by controlling mass escape and Lyapunov exponents.
Simple gradient descent algorithm escapes saddle points efficiently.
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
Geodesic loops escape from balls at a sublinear rate imply virtually abelian fundamental group.
Houdini finds high-dimensional saddle points under few constraints.
The Dirichlet random walk on manifolds has a positive escape rate if the cover is non-amenable.
We provide a theoretical algorithm for checking local optimality and escaping saddles at nondifferentiable points of empirical risks of two-layer ReLU networks. Our algorithm receives any parameter value and returns: local minimum, second-order stationary point, or a strict descent direction. The presence of data p…
HA-SME models SGD dynamics with Hessian info for better escaping behaviors.
New result on group actions in CAT(0) spaces with vanishing escape rate.
We solve the escape problem for the Heston random diffusion model. We obtain exact expressions for the survival probability (which ammounts to solving the complete escape problem) as well as for the mean exit time. We also average the volatility in order to work out the problem for the return alone regardless volatilit…
We study the mean escape time in a market model with stochastic volatility. The process followed by the volatility is the Cox Ingersoll and Ross process which is widely used to model stock price fluctuations. The market model can be considered as a generalization of the Heston model, where the geometric Brownian motion…
This paper proposes a new global optimization algorithm using deep learning.
This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i.e., it is almost "dimension-free"). The convergence rate of this procedure matches the well-known convergence rate of gradient descent to…
Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…
New algorithms improve sampling from Bayesian deep learning models.
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only stochastic gradien…
The roundworm C. elegans exhibits robust escape behavior in response to rapidly rising temperature. The behavior lasts for a few seconds, shows history dependence, involves both sensory and motor systems, and is too complicated to model mechanistically using currently available knowledge. Instead we model the process p…
ConViT combines CNN and ViT strengths, improving image classification.
New metrics help predict Brownian motion on surfaces and higher dimensions.
Quantum annealers aim at solving non-convex optimization problems by exploiting cooperative tunneling effects to escape local minima. The underlying idea consists in designing a classical energy function whose ground states are the sought optimal solutions of the original optimization problem and add a controllable qua…
A statistical analysis of financial, economic, and demographic indicators performed by the authors demonstrates (1) that the main countries of East Africa (Uganda, Kenya, and Tanzania) have not escaped the Malthusian Trap yet; (2) that this countries are not likely to follow the "North African path" and to achieve this…
Two-layer networks learn hard GLMs with SGD in high dimensions.
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although trai…
New -step policy gradient method avoids local optima in restricted policy classes.
The expectation-maximization (EM) algorithm has been widely used in minimizing the negative log likelihood (also known as cross entropy) of mixture models. However, little is understood about the goodness of the fixed points it converges to. In this paper, we study the regions where one component is missing in two-comp…