arXiv research

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

A new algorithm trains deep neural networks by adding neurons greedily.

DLNs dynamics change with variance, leading to saddle-to-saddle training phases.

New saddle network architectures preserve convex-concave geometry in optimization problems.

The paper studies neural networks' convergence near origin and saddle points.

SGD in DLNs reveals feature learning dynamics.

SGD learns neural networks with a complexity measure called leap.

Recent years have seen a growing interest in understanding deep neural networks from an optimization perspective. It is understood now that converging to low-cost local minima is sufficient for such models to become effective in practice. However, in this work, we propose a new hypothesis based on recent theoretical fi…

DEO uses gradient information to escape saddle points in neural networks.

New method solves saddle-point problems faster than existing methods.

The paper analyzes neural network dynamics after weights escape the origin.

This study explains gradient flow dynamics in neural networks for small initialisation.

A distributed optimization method solves saddle point problems with strong concavity and convexity.

Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…

We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our re…

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these l…

New method simplifies optimization landscapes by transforming saddle points.

New theory shows predictive coding makes learning landscape easier to navigate.

AGF explains feature learning in neural networks through alternating steps.

A new method trains physics-constrained neural networks more efficiently.

We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one ne…

In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015. With no unrealistic assumption, we first prove the following statements for the squared loss function of deep linear neural networks with any depth and any widths…

Gradient-based methods find saddle points, not critical points, in neural networks.

New method for LVEBMs using saddle-point optimization and Langevin updates.

Study finds saddle connections on random surfaces follow Poisson distribution.

Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.

Study shows saddle connection graph's geometry and quasi-isometry properties.

Algorithm classifies saddle-focus singularities in Hamiltonian systems.

A new optimisation method efficiently scales Hessian-vector products for neural networks.

Classifies Morse flows on 3-sphere with specific saddle connections.

APAC-Net solves high-dimensional stochastic MFGs using neural networks.

Study saddle connections on hyperelliptic surfaces, finding growth rates.

We extend asymptotic formulas for saddle connections on translation surfaces.

In this paper we study flows $\varphi:M\times\mathbb{R}\longrightarrow M$ having an isolated non-saddle set. We see that the complexity of the region of influence of an isolated non-saddle set $K$ depends on the way in which $K$ sits on the phase space at the cohomological level. We construct flows in surfaces having i…

The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima. In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, …

To every half-translation surface, we associate a saddle connection graph, which is a subgraph of the arc graph. We prove that every isomorphism between two saddle connection graphs is induced by an affine homeomorphism between the underlying half-translation surfaces. We also investigate the automorphism group of the …

Study precise rates of horizontal gap shrinkage on generic translation surfaces.

For a half-translation surface (S,q), the associated saddle connection complex A(S,q) is the simplicial complex where vertices are the saddle connections on (S,q), with simplices spanned by sets of pairwise disjoint saddle connections. This complex can be naturally regarded as an induced subcomplex of the arc complex. …

Golden L surface has unbounded bunching of saddle connections

FeDualEx tackles saddle point optimization in federated learning with composite objectives.

Saddle-point optimization problems are an important class of optimization problems with applications to game theory, multi-agent reinforcement learning and machine learning. A majority of the rich literature available for saddle-point optimization has focused on the offline setting. In this paper, we study nonstationar…

Study geometric properties of loss functions to understand neural network performance.

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

Although adversarial examples and model robustness have been extensively studied in the context of linear models and neural networks, research on this issue in tree-based models and how to make tree-based models robust against adversarial examples is still limited. In this paper, we show that tree based models are also…

Bounds on saddle connections on flat spheres with conical singularities.

In this paper, we focus on solving a class of constrained non-convex non-concave saddle point problems in a decentralized manner by a group of nodes in a network. Specifically, we assume that each node has access to a summand of a global objective function and nodes are allowed to exchange information only with their n…

We consider discriminative dictionary learning in a distributed online setting, where a network of agents aims to learn a common set of dictionary elements of a feature space and model parameters while sequentially receiving observations. We formulate this problem as a distributed stochastic program with a non-convex o…

We are concerned with the saddle solutions of the Allen-Cahn equation constructed by Cabré and Terra \cite{C,C2} in $\mathbb{R}^{2m}$. These solutions vanish precisely on the Simons cone. The existence and uniqueness of saddle solution are shown in \cite{C,C2,C1}. Regarding the stab…