SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.
DLNs dynamics change with variance, leading to saddle-to-saddle training phases.
problem Understanding the dynamics of DLNs with varying initialization variance.
method Analyzing the phase transition of DLNs' dynamics as variance changes.
result Gradient descent visits a sequence of saddles, reaching a sparse global minimum.
Study dynamics and topology of flows near non-saddle sets or W-sets.
problem Understanding the dynamics and topology of flows near specific invariant sets.
method Cohomological relations and global properties analysis.
result Dynamical classification of surfaces and robustness of non-saddle-sets.
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value. Equivalence of convex optimization, saddle-point problems, and variational inequalities is a well-established concept. The variational inequality (VI) is a static problem which is studied under dynamical settings using a framework called the projected dynamical system, whose stationary points coincide with the static s…
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
SGD learns neural networks with a complexity measure called leap.
problem Time complexity of SGD learning on neural networks.
method Introduced a complexity measure called leap, proved conjecture for Gaussian data, and showed saddle-to-saddle dynamics.
result Proved a conjecture about the time complexity of learning functions with low-dimensional support.
This study explains gradient flow dynamics in neural networks for small initialisation.
problem Understanding the training dynamics of neural networks for small initialisation.
method Analysis of gradient flow dynamics for one-hidden layer ReLU networks with orthogonal inputs.
result Gradient flow converges to zero loss and characterizes implicit bias towards minimum variation norm.
Active learning reduces SP calculations by 90%.
problem Efficiently calculating saddle points in energy functions.
method Active learning framework with GPR and GAD.
result Significant reduction in the number of expensive evaluations.
The paper analyzes neural network dynamics after weights escape the origin.
problem Understanding gradient flow dynamics of neural networks after the origin.
method Analyzes gradient flow of homogeneous neural networks with locally Lipschitz gradients.
result Characterizes the first saddle point encountered after escaping the origin.
Translation surfaces with poles correspond to meromorphic differentials on compact Riemann surfaces. They appear in compactifications of strata of the moduli space of Abelian differentials and in the study of stability conditions. Such structures have different geometrical and dynamical properties than usual translatio…
Paper defines saddle points in asymmetric Dynkin games using martingale theory.
problem Tackles saddle point conditions in asymmetric Dynkin games with partial information.
method Uses martingale theory to identify super and submartingales related to equilibrium payoffs.
result Characterizes saddle point strategies in terms of equilibrium payoffs' dynamics and Doob-Meyer decompositions.
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…
We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical s…
Paper analyzes Transformer learning dynamics, proving benign landscape for in-context learning.
problem Understanding how Transformers learn in context with nonlinear features.
method Mean-field and two-timescale analysis of Transformer dynamics, proving nonconvex but benign landscape.
result Proves mean-field dynamics avoid saddle points, leading to improved optimization.
WSFN overcomes saddle points for non-convex functionals in Wasserstein space.
problem Minimizing non-convex functionals over the Wasserstein space with saddle point avoidance.
method WSFN is a second-order method that preconditions the Wasserstein gradient to avoid saddle points.
result WSFN escapes saddle regions and reaches a global minimizer in polynomial time.
Flat surfaces that correspond to k-differentials on compact Riemann surfaces are of finite area provided there is no pole of order k or higher. We denote by \textit{flat surfaces with poles of higher order} those surfaces with flat structures defined by a k-differential with at least one pole of order at least $k…
New ODE models show saddle-point optimization methods converge differently, with last-iterate convergence for OGDA.
problem Analyzing convergence properties of saddle-point optimization methods.
method High-Resolution Differential Equations (HRDEs) to design differential equation models for saddle-point optimization methods.
result HRDEs reveal last-iterate convergence for Optimistic Gradient Descent Ascent (OGDA) in bilinear games.
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…
GenFlow optimizes faster, avoiding saddle points in fixed time.
problem Designing efficient optimization algorithms for convex and non-convex functions.
method Introduces GenFlow and momentum variants with fixed-time convergence guarantees.
result GenFlow and momentum variants converge to optimal solutions in fixed time for PL functions and evade saddle points uniformly.
Researchers compute gap distributions for saddle connection directions on specific translation surfaces.
problem Computing gap distributions for saddle connection directions on translation surfaces.
method Translation to dynamical question of return times to a transversal under the horocycle flow.
result Gap distributions have support at 0 and quadratic tail decay.
Horizon saddle connections imply dense hyperbolic geodesics on dilation surfaces.
problem Characterize dilation surfaces with dense hyperbolic geodesics.
method Analyzing saddle connections and directional flow properties.
result Dilation surfaces with horizon saddle connections have dense hyperbolic geodesics.
Gradient flow solves multi-index regression for high-dimensional Gaussian data.
problem Learning multi-index functions from high-dimensional Gaussian data.
method Two-timescale algorithm with non-parametric link function learning.
result Global convergence of Grassmannian population gradient flow dynamics.
New method for LVEBMs using saddle-point optimization and Langevin updates.
problem Expressive generative modeling of latent variables with hidden structure.
method Reformulate LVEBM training as a saddle problem, using Langevin updates and gradient flows.
result Proves existence and convergence of the algorithm under standard assumptions, with improved ELBO bounds.
DEO uses gradient information to escape saddle points in neural networks.
problem Training deep neural networks struggles with flat regions and saddle points.
method Dimer-Enhanced Optimization (DEO) uses gradient information to estimate curvature and escape saddle points.
result DEO improves training efficiency and performance compared to standard first-order methods.
AGF explains feature learning in neural networks through alternating steps.
problem Understanding what features neural networks learn and how they learn them.
method AGF is an algorithmic framework that approximates the dynamics of feature learning in two-layer networks.
result AGF provides a unified framework to understand feature learning in neural networks, matching experimental results across various architectures.
Develops path integral for spiked tensor model dynamics.
problem Dynamics of spiked tensor model with random initial conditions.
method Path integral approach applied to partial differential equations.
result Large-N saddle point equations dominated by melonic diagrams. Efficient algorithm converges to Nash equilibrium in bilinear problems with bandit feedback.
problem Learning dynamics in bilinear saddle-point problems with bandit feedback.
method Uncoupled learning algorithm combining experimental design and FTRL with a tailored regularizer.
result Last-iterate convergence rate of ildeO(T−1/4) in high probability. We consider discriminative dictionary learning in a distributed online setting, where a network of agents aims to learn a common set of dictionary elements of a feature space and model parameters while sequentially receiving observations. We formulate this problem as a distributed stochastic program with a non-convex o…
OKRidge solves sparse ridge regression problems for nonlinear systems.
problem Identifying sparse governing equations for nonlinear dynamical systems.
method OKRidge algorithm using saddle point formulation and ADMM-based approach with efficient proximal operators.
result OKRidge achieves provable optimality with significantly faster run times than Gurobi.
Gradient descent proves global convergence for 4-layer matrix factorization.
problem Global convergence of gradient descent on four-layer matrix factorization under random initialization.
method New techniques to show saddle-avoidance properties and extend eigenvalue theories.
result Polynomial-time global convergence guarantee for randomly initialized gradient descent on four-layer matrix factorization.
Paper shows equivalence between MM and PH for n-D Morse functions.
problem Relationship between Mathematical Morphology and Persistent Homology.
method Examined pairing of extrema in Morse functions using dynamics and persistence.
result Equivalence proven between dynamics and persistence on n-D Morse functions.
The altenating knots, links and twists projected on the S_2 sphere are identified with the phase Space of a Hamiltonian dynamic system of one degree of freedom. The saddles of the system correspond to the crossing points, the edges, to the stable and unstable manifolds, connecting the saddles. Each facxe is then orient…
This short survey illustrates the ideas of Teichmuller dynamics. As a model application we consider the asymptotic topology of generic geodesics on a "flat" surface and count closed geodesics and saddle connections. This survey is based on the joint papers with A.Eskin and H.Masur and with M.Kontsevich.
We use martingale and stochastic analysis techniques to study a continuous-time optimal stopping problem, in which the decision maker uses a dynamic convex risk measure to evaluate future rewards. We also find a saddle point for an equivalent zero-sum game of control and stopping, between an agent (the "stopper") who c…
Study finds saddle connections on random surfaces follow Poisson distribution.
problem Distribution of saddle connections on random translation surfaces.
method Analysis of saddle connections on surfaces of large genus.
result Number of saddle connections in given lengths converges to Poisson distribution.
New algorithms for online learning without boundedness or Lipschitz loss assumptions.
problem Online learning with unbounded domains and non-Lipschitz losses.
method Developed an algorithm with a specific regret bound and used it for saddle-point optimization.
result First algorithm achieving non-trivial dynamic regret in an unbounded domain for non-Lipschitz losses.
Gradient descent trains both layers of a ReLU network to fit a linear model.
problem Training dynamics of a ReLU network to fit a linear target function.
method Jointly training both layers of a one-hidden-layer ReLU network in a realizable setting with Gaussian inputs and labels.
result Gradient descent from a small random initialization converges to a global minimizer at a linear rate with optimal sample complexity.
This paper explains why Adam generalizes worse than SGD by analyzing its components.
problem Understanding why Adam generalizes worse than Stochastic Gradient Descent (SGD).
method Diffusion theoretical framework to disentangle the effects of Adaptive Learning Rate and Momentum.
result Adaptive Learning Rate helps escape saddle points but not select flat minima, while Momentum provides a drift effect to help pass through saddle points.
Study shows saddle connection graph's geometry and quasi-isometry properties.
problem Characterize the geometry and quasi-isometry of saddle connection graphs.
method Proved 4-hyperbolicity and uniform quasi-isometry to a tree, used generalised unicorn paths.
result Saddle connection graph is not quasi-isometrically rigid and its boundary is straight foliations.
Algorithm classifies saddle-focus singularities in Hamiltonian systems.
problem Classifying nondegenerate saddle-focus singularities in integrable Hamiltonian systems.
method Developed an algorithm based on semi-local equivalence to represent singularities as almost direct products.
result Obtained complete lists of saddle-focus singularities of complexities 1, 2, and 3.
Classifies Morse flows on 3-sphere with specific saddle connections.
problem Classifying Morse-Smale flows on a 3-sphere with specific saddle connections.
method Used generalized Heegaard diagrams (Pr-diagrams) to classify flows.
result Found all possible, up to homeomorphism, ways to embed two circles in a 2-sphere with no more than 10 points of transversal intersection.
New algorithms improve sampling from Bayesian deep learning models.
problem Sampling from the posterior of deep neural networks is inefficient.
method Adaptive SGMCMC algorithms with biased drift.
result Proposed algorithms significantly outperform existing methods.
Study saddle connections on hyperelliptic surfaces, finding growth rates.
problem Count saddle connections on hyperelliptic surfaces without interior intersections.
method Used horocycle renormalization to prove lower bound growth rate.
result Found saddle connections satisfy L(logL)d−2 growth rate. We extend asymptotic formulas for saddle connections on translation surfaces.
problem Counting saddle connections on translation surfaces with large genus.
method Recursive formulas and asymptotic analysis for all strata and multiplicities.
result Asymptotics for all saddle connections on translation surfaces of growing genus.
In this paper we study flows φ:M×R⟶M having an isolated non-saddle set. We see that the complexity of the region of influence of an isolated non-saddle set K depends on the way in which K sits on the phase space at the cohomological level. We construct flows in surfaces having i…
We study the online saddle point problem, an online learning problem where at each iteration a pair of actions need to be chosen without knowledge of the current and future (convex-concave) payoff functions. The objective is to minimize the gap between the cumulative payoffs and the saddle point value of the aggregate …
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
problem Spurious critical points on the boundary of low-rank matrix manifold.
method Riemannian gradient descent with dynamical low-rank approximation and rescaled gradient flow.
result Riemannian gradient descent escapes some spurious critical points on the boundary of the manifold.