The paper analyzes neural network dynamics after weights escape the origin.
problem Understanding gradient flow dynamics of neural networks after the origin.
method Analyzes gradient flow of homogeneous neural networks with locally Lipschitz gradients.
result Characterizes the first saddle point encountered after escaping the origin.
Study of SGD with state-dependent noise, improving escape from local minima.
problem Understanding and improving the dynamics of SGD in non-convex optimization.
method Formal study on SGD with state-dependent noise, proposing power-law dynamic with state-dependent diffusion.
result Power-law dynamic can escape from sharp minima exponentially faster than flat minima.
Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this …
The roundworm C. elegans exhibits robust escape behavior in response to rapidly rising temperature. The behavior lasts for a few seconds, shows history dependence, involves both sensory and motor systems, and is too complicated to model mechanistically using currently available knowledge. Instead we model the process p…
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value. HA-SME models SGD dynamics with Hessian info for better escaping behaviors.
problem Capturing the escaping behaviors of SGD from stationary points.
method HA-SME, a novel SDE with Hessian info in drift and diffusion.
result HA-SME achieves best approximation error and recovers SGD dynamics for quadratics.
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…
SGD transitions between maxima and minima with varying time scales.
problem Understanding SGD's behavior near critical points in noisy landscapes.
method Analyzing SGD convergence and escape dynamics in 1D landscapes with infinite- and finite-variance noise.
result SGD reliably moves to the basin's minimum unless close to a local maximum, where it can linger.
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
problem Spurious critical points on the boundary of low-rank matrix manifold.
method Riemannian gradient descent with dynamical low-rank approximation and rescaled gradient flow.
result Riemannian gradient descent escapes some spurious critical points on the boundary of the manifold.
One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
problem Understanding how noise geometry affects the performance of stochastic gradient descent.
method Developed two metrics to quantify noise alignment strength and analyzed their effects on loss and subspace projection dynamics.
result Noise geometry can be used to guarantee alignment under certain conditions, aiding SGD's ability to escape from sharp minima.
Paper learns Koopman operator from sparse data, escaping function space constraints.
problem Learning Koopman operator from non-closed function spaces.
method Operator stochastic approximation algorithm using conditional mean embeddings (CME).
result Online sparse learning algorithm with trajectory-based sampling guarantees.
Study shows how anisotropic data affects learning dynamics in phase retrieval.
problem Understanding learning dynamics in phase retrieval with anisotropic Gaussian inputs.
method Developed a tractable reduction to reveal a three-phase trajectory and derived scaling laws.
result Found that anisotropy leads to a three-phase trajectory: fast escape, slow convergence, and spectral-tail learning.
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…
Two-layer networks learn hard GLMs with SGD in high dimensions.
problem Learning hard generalized linear models with SGD in high-dimensional settings.
method Reduction of SGD dynamics to a stochastic process in lower dimensions, focusing on the role of stochasticity.
result Overparameterization enhances convergence by a constant factor, suggesting minimal role of stochasticity.
Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.
problem Understanding the fundamental groups of open manifolds with nonnegative Ricci curvature.
method Analyzing the escape rate of minimal geodesic loops and relating it to the fundamental group's properties.
result If an open manifold has a small escape rate, its fundamental group is virtually abelian.
We study the SL2(R)-action on the moduli space of (triangulable) dilation tori with one boundary component. We prove that every orbit is either closed or dense, and that every orbit of the Teichmuller flow escapes to infinity.
Algorithm learns stochastic system dynamics from data.
problem Recovering interpretable symbolic expressions for stochastic systems.
method Data-driven, trajectory averaging, drift-informed correction.
result Recover coefficients and densities to within 5% and 0.01 in total variation, respectively.
New algorithms improve sampling from Bayesian deep learning models.
problem Sampling from the posterior of deep neural networks is inefficient.
method Adaptive SGMCMC algorithms with biased drift.
result Proposed algorithms significantly outperform existing methods.
SGD's escape rate depends on log loss barrier, not linear loss barrier.
problem Understanding the escape rate of SGD from local minima.
method Derived a stochastic differential equation (SDE) with additive noise from SGD's multiplicative noise property.
result The log loss barrier determines the escape rate of SGD, not the linear loss barrier.
We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…
Classifies conformal transformations in spacetimes without observer horizons.
problem Understanding conformal transformations in spacetimes without observer horizons.
method Proves classification of conformal transformations into two types: escaping and non-escaping.
result Conformal transformations of Einstein's static universe are classified.
We shortly review the statistical properties of the escape times, or hitting times, for stock price returns by using different models which describe the stock market evolution. We compare the probability function (PF) of these escape times with that obtained from real market data. Afterwards we analyze in detail the ef…
Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.
problem Finding safe zones in policy Markov Decision Processes to limit trajectory escape.
method Bi-criteria approximation learning algorithm with polynomial sample complexity.
result Achieves almost 2 approximation for both escape probability and safe zone size.
Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
New algorithm helps escape saddle points in optimization problems.
problem Optimizing smooth non-convex functions to avoid saddle points.
method Perturbed Saddle-escape Descent (PSD) algorithm with explicit constants.
result PSD finds approximate second-order stationary points efficiently.
The paper proves Zimmer's conjecture for non-uniform lattices by controlling mass escape and Lyapunov exponents.
problem Proving Zimmer's conjecture for non-uniform lattices in higher-rank semisimple Lie groups.
method Establishes finiteness of low-dimensional actions, introduces novel techniques to control mass escape and Lyapunov exponents.
result Proves Zimmer's conjecture for many non-uniform lattices, improving previous results.
New methods help escape strict saddle points in nonsmooth optimization.
problem Escaping strict saddle points in nonsmooth optimization.
method An inexact stochastically perturbed gradient method applied to the Moreau envelope.
result A variety of algorithms for nonsmooth optimization can efficiently escape strict saddle points of the Moreau envelope.
Geodesic loops escape from balls at a sublinear rate imply virtually abelian fundamental group.
problem Understanding fundamental groups of open manifolds with nonnegative Ricci curvature.
method Generalizing the Cheeger-Gromoll splitting theorem to sublinear escape rates.
result Fundamental groups of open manifolds with nonnegative Ricci curvature are virtually abelian if geodesic loops escape sublinearly.
Houdini finds high-dimensional saddle points under few constraints.
problem Escaping from saddle points in high-dimensional spaces with constraints.
method Gradient descent methods under logarithmic inequality constraints.
result Polynomial time algorithms for escaping saddle points under constraints.
Deep learning dynamics exhibit anomalous superdiffusion initially, aiding escape from local minima.
problem Understanding the dynamics of learning in deep neural networks.
method Novel analysis of SGD dynamics and loss landscape structure.
result SGD exhibits anomalous superdiffusion initially, transitioning to subdiffusion as learning progresses.
Authors provide counterexamples to Weinstein conjecture in 3D.
problem Weinstein conjecture in 3D contact geometry.
method Construction of b-contact manifolds with specific orbits.
result Counterexamples to Weinstein conjecture in 3D.
The Dirichlet random walk on manifolds has a positive escape rate if the cover is non-amenable.
problem Analyzing the stochastic behavior of Dirichlet random walks on manifolds.
method Defining a recursive process on Galoisian covers and proving a theorem about the escape rate.
result The escape rate is positive if and only if the cover is non-amenable.
Rapid mixing of Langevin dynamics on Riemannian manifolds
problem Mixing time of Langevin dynamics on Riemannian manifolds
method Relation between Langevin processes in domain and image
result Achievable polynomial mixing times
New result on group actions in CAT(0) spaces with vanishing escape rate.
problem Understanding group actions with vanishing escape rate on CAT(0) spaces.
method Equivariant μ-harmonic map proof. result Existence of a flat subspace invariant under the action of Γ. We solve the escape problem for the Heston random diffusion model. We obtain exact expressions for the survival probability (which ammounts to solving the complete escape problem) as well as for the mean exit time. We also average the volatility in order to work out the problem for the return alone regardless volatilit…
We study the mean escape time in a market model with stochastic volatility. The process followed by the volatility is the Cox Ingersoll and Ross process which is widely used to model stock price fluctuations. The market model can be considered as a generalization of the Heston model, where the geometric Brownian motion…
This paper proposes a new global optimization algorithm using deep learning.
problem Developing efficient algorithms for global optimization of non-convex functions.
method Two-phase approach: minimization phase with model-driven deep learning, escaping phase with reinforcement learning.
result The proposed algorithm significantly outperforms classical optimization methods and handles ill-posed functions.
New pricing algorithm learns demand curves and optimizes prices in dynamic markets.
problem Dynamic pricing in markets with incomplete demand information and shifting conditions.
method Actor-Critic Information-Directed Pricing (ACIDP) using IDS algorithms and auditing procedures.
result ACIDP outperforms UCB and TS in market environment shifts.
DEO uses gradient information to escape saddle points in neural networks.
problem Training deep neural networks struggles with flat regions and saddle points.
method Dimer-Enhanced Optimization (DEO) uses gradient information to estimate curvature and escape saddle points.
result DEO improves training efficiency and performance compared to standard first-order methods.
We prove Zimmer's conjecture for C2 actions by finite-index subgroups of SL(m,Z) provided m>3. The method utilizes many ingredients from our earlier proof of the conjecture for actions by cocompact lattices in SL(m,R) but new ideas are needed to overcome the lack of compactn…
Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
problem Escaping local minima in loss landscapes.
method Hill-ADAM alternates between minimizing and maximizing error to explore the loss space.
result Hill-ADAM finds the global minimum state in loss landscapes.
Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…
A new method helps escape saddle points in non-convex optimization.
problem Escaping saddle points in non-convex optimization problems.
method CNC-SCSG method using a separate SGD step to help escape from strict saddle points.
result The method converges to a second-order stationary point with a rate of O(ε−2log(1/ε)). Study shows gain-loss asymmetry in stock indices using a q-spin Potts model.
problem Understanding the dynamics of stock indices in complex markets.
method Developed a q-spin Potts model to represent stock market dynamics.
result Observed a self-organized gain-loss asymmetry in stock indices.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
We study the Stochastic Gradient Langevin Dynamics (SGLD) algorithm for non-convex optimization. The algorithm performs stochastic gradient descent, where in each step it injects appropriately scaled Gaussian noise to the update. We analyze the algorithm's hitting time to an arbitrary subset of the parameter space. Two…