Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value. DLNs dynamics change with variance, leading to saddle-to-saddle training phases.
problem Understanding the dynamics of DLNs with varying initialization variance.
method Analyzing the phase transition of DLNs' dynamics as variance changes.
result Gradient descent visits a sequence of saddles, reaching a sparse global minimum.
Study finds saddle connections on random surfaces follow Poisson distribution.
problem Distribution of saddle connections on random translation surfaces.
method Analysis of saddle connections on surfaces of large genus.
result Number of saddle connections in given lengths converges to Poisson distribution.
Study shows saddle connection graph's geometry and quasi-isometry properties.
problem Characterize the geometry and quasi-isometry of saddle connection graphs.
method Proved 4-hyperbolicity and uniform quasi-isometry to a tree, used generalised unicorn paths.
result Saddle connection graph is not quasi-isometrically rigid and its boundary is straight foliations.
Algorithm classifies saddle-focus singularities in Hamiltonian systems.
problem Classifying nondegenerate saddle-focus singularities in integrable Hamiltonian systems.
method Developed an algorithm based on semi-local equivalence to represent singularities as almost direct products.
result Obtained complete lists of saddle-focus singularities of complexities 1, 2, and 3.
Classifies Morse flows on 3-sphere with specific saddle connections.
problem Classifying Morse-Smale flows on a 3-sphere with specific saddle connections.
method Used generalized Heegaard diagrams (Pr-diagrams) to classify flows.
result Found all possible, up to homeomorphism, ways to embed two circles in a 2-sphere with no more than 10 points of transversal intersection.
Study saddle connections on hyperelliptic surfaces, finding growth rates.
problem Count saddle connections on hyperelliptic surfaces without interior intersections.
method Used horocycle renormalization to prove lower bound growth rate.
result Found saddle connections satisfy L(logL)d−2 growth rate. SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.
We extend asymptotic formulas for saddle connections on translation surfaces.
problem Counting saddle connections on translation surfaces with large genus.
method Recursive formulas and asymptotic analysis for all strata and multiplicities.
result Asymptotics for all saddle connections on translation surfaces of growing genus.
In this paper we study flows φ:M×R⟶M having an isolated non-saddle set. We see that the complexity of the region of influence of an isolated non-saddle set K depends on the way in which K sits on the phase space at the cohomological level. We construct flows in surfaces having i…
To every half-translation surface, we associate a saddle connection graph, which is a subgraph of the arc graph. We prove that every isomorphism between two saddle connection graphs is induced by an affine homeomorphism between the underlying half-translation surfaces. We also investigate the automorphism group of the …
Study precise rates of horizontal gap shrinkage on generic translation surfaces.
problem Understanding precise decay rates of horizontal gaps in translation surfaces.
method Analyzing saddle connections and their angles on translation surfaces.
result Obtained precise decay rates for the difference in angle between almost horizontal saddle connections.
For a half-translation surface (S,q), the associated saddle connection complex A(S,q) is the simplicial complex where vertices are the saddle connections on (S,q), with simplices spanned by sets of pairwise disjoint saddle connections. This complex can be naturally regarded as an induced subcomplex of the arc complex. …
Golden L surface has unbounded bunching of saddle connections
problem Unbounded bunching of saddle connections on translation surfaces
method Translation surface with golden ratio
result Every positive integer K has a ball containing at least K saddle connection periods
FeDualEx tackles saddle point optimization in federated learning with composite objectives.
problem Saddle point optimization with constraints and non-smooth regularization in federated learning.
method Federated Dual Extrapolation (FeDualEx) algorithm for saddle point optimization and composite objectives.
result FeDualEx effectively solves saddle point optimization problems with composite objectives in federated learning.
Saddle-point optimization problems are an important class of optimization problems with applications to game theory, multi-agent reinforcement learning and machine learning. A majority of the rich literature available for saddle-point optimization has focused on the offline setting. In this paper, we study nonstationar…
Bounds on saddle connections on flat spheres with conical singularities.
problem Counting saddle connections on flat spheres with conical singularities.
method Geometry of immersed disks and explicit upper bounds.
result Explicit upper bounds on the number and lengths of saddle connections.
We are concerned with the saddle solutions of the Allen-Cahn equation constructed by Cabré and Terra \cite{C,C2} in R2m. These solutions vanish precisely on the Simons cone. The existence and uniqueness of saddle solution are shown in \cite{C,C2,C1}. Regarding the stab…
Study dynamics and topology of flows near non-saddle sets or W-sets.
problem Understanding the dynamics and topology of flows near specific invariant sets.
method Cohomological relations and global properties analysis.
result Dynamical classification of surfaces and robustness of non-saddle-sets.
We show that area minimizing polyhedral surfaces are saddle.
The study shows how to measure translation surfaces with short saddle connections.
problem Measuring the probability of surfaces with short saddle connections.
method Using the multi-scale compactification of strata and algebraicity results.
result Proves strong regularity for invariant measures on translation surfaces.
Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
Researchers create new triply periodic minimal surfaces by gluing saddle towers.
problem Creating triply periodic minimal surfaces without symmetry constraints.
method Gluing Karcher-Scherk saddle towers with phase differences and balancing under vertical interaction.
result Expands known triply periodic minimal surfaces into new 5-parameter families.
WSFN overcomes saddle points for non-convex functionals in Wasserstein space.
problem Minimizing non-convex functionals over the Wasserstein space with saddle point avoidance.
method WSFN is a second-order method that preconditions the Wasserstein gradient to avoid saddle points.
result WSFN escapes saddle regions and reaches a global minimizer in polynomial time.
New methods help escape strict saddle points in nonsmooth optimization.
problem Escaping strict saddle points in nonsmooth optimization.
method An inexact stochastically perturbed gradient method applied to the Moreau envelope.
result A variety of algorithms for nonsmooth optimization can efficiently escape strict saddle points of the Moreau envelope.
New insights into matrix factorization show strict saddles have bounded eigenvalues.
problem Understanding the nature of critical points in matrix factorization.
method Analyzing orbits of critical points under the general linear group and identifying canonical points.
result Minimum eigenvalue of strict saddles is not uniformly bounded below zero.
We prove an analog of the Schoen-Yau univalentness theorem for saddle maps between discs.
A new algorithm trains deep neural networks by adding neurons greedily.
problem Training deep neural networks efficiently and effectively.
method Neuron Pursuit (NP) algorithm, which alternates between neuron addition and loss minimization.
result The algorithm can train deep neural networks efficiently and effectively.
New method solves saddle-point problems faster than existing methods.
problem Large-scale saddle-point problems in optimization.
method Sequential subspace optimization with proximal regularization.
result Significantly better convergence compared to first-order methods.
A new method helps escape saddle points in non-convex optimization.
problem Escaping saddle points in non-convex optimization problems.
method CNC-SCSG method using a separate SGD step to help escape from strict saddle points.
result The method converges to a second-order stationary point with a rate of O(ε−2log(1/ε)). New saddle network architectures preserve convex-concave geometry in optimization problems.
problem Optimization models with convex x and concave y components.
method Structured separable decomposition and saddle network architectures.
result Proven one-dimensional approximation theorem and high accuracy on various test functions.
Houdini finds high-dimensional saddle points under few constraints.
problem Escaping from saddle points in high-dimensional spaces with constraints.
method Gradient descent methods under logarithmic inequality constraints.
result Polynomial time algorithms for escaping saddle points under constraints.
This paper extends Newton's method to distributed learning, avoiding saddle points and handling Byzantine workers.
problem Avoiding saddle points in distributed non-convex optimization, especially in the presence of Byzantine workers.
method Extends cubic-regularized Newton method to distributed framework, addressing communication bottlenecks and Byzantine attacks.
result The method achieves improved iteration complexity compared to first-order methods, with a 25% improvement in experiments.
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
Gradient-like flows on certain manifolds restrict saddle Morse indices to 1 or n-1.
problem Restricting Morse indices of saddles in gradient-like flows.
method Analyzing invariant manifolds and their intersections for gradient-like flows.
result Morse indices of saddles are either 1 or n-1, no other indices possible.
Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…
We introduce a geometrically transparent strict saddle property for nonsmooth functions. This property guarantees that simple proximal algorithms on weakly convex problems converge only to local minimizers, when randomly initialized. We argue that the strict saddle property may be a realistic assumption in applications…
New ODE models show saddle-point optimization methods converge differently, with last-iterate convergence for OGDA.
problem Analyzing convergence properties of saddle-point optimization methods.
method High-Resolution Differential Equations (HRDEs) to design differential equation models for saddle-point optimization methods.
result HRDEs reveal last-iterate convergence for Optimistic Gradient Descent Ascent (OGDA) in bilinear games.
Translation surfaces with poles correspond to meromorphic differentials on compact Riemann surfaces. They appear in compactifications of strata of the moduli space of Abelian differentials and in the study of stability conditions. Such structures have different geometrical and dynamical properties than usual translatio…
Extends saddle-point method for large-time volatility smiles.
problem Analyzing large-time volatility smiles in financial models.
method Saddle-point approach to derive large-time model-implied volatility smiles.
result Provides theoretical foundation and wide class of arbitrage-free parametrizations.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
Paper defines saddle points in asymmetric Dynkin games using martingale theory.
problem Tackles saddle point conditions in asymmetric Dynkin games with partial information.
method Uses martingale theory to identify super and submartingales related to equilibrium payoffs.
result Characterizes saddle point strategies in terms of equilibrium payoffs' dynamics and Doob-Meyer decompositions.
Recent years have seen a growing interest in understanding deep neural networks from an optimization perspective. It is understood now that converging to low-cost local minima is sufficient for such models to become effective in practice. However, in this work, we propose a new hypothesis based on recent theoretical fi…
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…
Analytic saddle spheres in S^3 are equators.
problem Characterizing saddle-shaped minimal surfaces in 3-sphere.
method Purely geometric approach, no PDE imposed.
result Analytic saddle spheres in S^3 are equators.
SGD learns neural networks with a complexity measure called leap.
problem Time complexity of SGD learning on neural networks.
method Introduced a complexity measure called leap, proved conjecture for Gaussian data, and showed saddle-to-saddle dynamics.
result Proved a conjecture about the time complexity of learning functions with low-dimensional support.
GenFlow optimizes faster, avoiding saddle points in fixed time.
problem Designing efficient optimization algorithms for convex and non-convex functions.
method Introduces GenFlow and momentum variants with fixed-time convergence guarantees.
result GenFlow and momentum variants converge to optimal solutions in fixed time for PL functions and evade saddle points uniformly.
New algorithm helps escape saddle points in optimization problems.
problem Optimizing smooth non-convex functions to avoid saddle points.
method Perturbed Saddle-escape Descent (PSD) algorithm with explicit constants.
result PSD finds approximate second-order stationary points efficiently.