arXiv research

New insights into matrix factorization show strict saddles have bounded eigenvalues.

We introduce a geometrically transparent strict saddle property for nonsmooth functions. This property guarantees that simple proximal algorithms on weakly convex problems converge only to local minimizers, when randomly initialized. We argue that the strict saddle property may be a realistic assumption in applications…

New methods help escape strict saddle points in nonsmooth optimization.

We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of init…

New theory shows predictive coding makes learning landscape easier to navigate.

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

WSFN overcomes saddle points for non-convex functionals in Wasserstein space.

New algorithm helps escape saddle points in optimization problems.

A new method helps escape saddle points in non-convex optimization.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

SGD avoids critical points on weakly convex functions.

We propose a general theory for studying the \xl{landscape} of nonconvex \xl{optimization} with underlying symmetric structures \tz{for a class of machine learning problems (e.g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}. In specific, we characterize the locations of stationary …

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

Under appropriate cooperation protocols and parameter choices, fully decentralized solutions for stochastic optimization have been shown to match the performance of centralized solutions and result in linear speedup (in the number of agents) relative to non-cooperative approaches in the strongly-convex setting. More re…

SGD converges almost surely in non-convex problems, avoiding saddle points and accelerating convergence.

Stochastic subgradient descent avoids critical points in definable functions.

Gradient descent trains both layers of a ReLU network to fit a linear model.

This work justifies neural collapse under MSE loss and analyzes the optimization landscape.

We prove a \emph{query complexity} lower bound on rank-one principal component analysis (PCA). We consider an oracle model where, given a symmetric matrix $M \in \mathbb{R}^{d \times d}$, an algorithm is allowed to make $T$ \emph{exact} queries of the form $w^{(i)} = Mv^{(i)}$ for $i \in \{1,\dots,T\}$, where $v^{(i)}$…

Generative Adversarial Networks have been shown to be powerful in generating content. To this end, they have been studied intensively in the last few years. Nonetheless, training these networks requires solving a saddle point problem that is difficult to solve and slowly converging. Motivated from techniques in the reg…

Paper explains neural collapse in neural networks using a new model.

We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with $M \geq 3$ components. Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical…

New tensor recovery method improves efficiency under strict complementarity.

We provide a theoretical algorithm for checking local optimality and escaping saddles at nondifferentiable points of empirical risks of two-layer ReLU networks. Our algorithm receives any parameter value and returns: local minimum, second-order stationary point, or a strict descent direction. The presence of $M$ data p…

Owing to their connection with generative adversarial networks (GANs), saddle-point problems have recently attracted considerable interest in machine learning and beyond. By necessity, most theoretical guarantees revolve around convex-concave (or even linear) problems; however, making theoretical inroads towards effici…

DLNs dynamics change with variance, leading to saddle-to-saddle training phases.

Study finds saddle connections on random surfaces follow Poisson distribution.

Study shows saddle connection graph's geometry and quasi-isometry properties.

Algorithm classifies saddle-focus singularities in Hamiltonian systems.

Classifies Morse flows on 3-sphere with specific saddle connections.

Study saddle connections on hyperelliptic surfaces, finding growth rates.

SGD in DLNs reveals feature learning dynamics.

We extend asymptotic formulas for saddle connections on translation surfaces.

In this paper we study flows $\varphi:M\times\mathbb{R}\longrightarrow M$ having an isolated non-saddle set. We see that the complexity of the region of influence of an isolated non-saddle set $K$ depends on the way in which $K$ sits on the phase space at the cohomological level. We construct flows in surfaces having i…

To every half-translation surface, we associate a saddle connection graph, which is a subgraph of the arc graph. We prove that every isomorphism between two saddle connection graphs is induced by an affine homeomorphism between the underlying half-translation surfaces. We also investigate the automorphism group of the …

Study precise rates of horizontal gap shrinkage on generic translation surfaces.

For a half-translation surface (S,q), the associated saddle connection complex A(S,q) is the simplicial complex where vertices are the saddle connections on (S,q), with simplices spanned by sets of pairwise disjoint saddle connections. This complex can be naturally regarded as an induced subcomplex of the arc complex. …

Golden L surface has unbounded bunching of saddle connections

We consider the minimization of non-convex functions that typically arise in machine learning. Specifically, we focus our attention on a variant of trust region methods known as cubic regularization. This approach is particularly attractive because it escapes strict saddle points and it provides stronger convergence gu…

FeDualEx tackles saddle point optimization in federated learning with composite objectives.

Saddle-point optimization problems are an important class of optimization problems with applications to game theory, multi-agent reinforcement learning and machine learning. A majority of the rich literature available for saddle-point optimization has focused on the offline setting. In this paper, we study nonstationar…

Bounds on saddle connections on flat spheres with conical singularities.

We are concerned with the saddle solutions of the Allen-Cahn equation constructed by Cabré and Terra \cite{C,C2} in $\mathbb{R}^{2m}$. These solutions vanish precisely on the Simons cone. The existence and uniqueness of saddle solution are shown in \cite{C,C2,C1}. Regarding the stab…

Study dynamics and topology of flows near non-saddle sets or W-sets.

We show that area minimizing polyhedral surfaces are saddle.

The study shows how to measure translation surfaces with short saddle connections.