arXiv research

This work examines the stability of GD and SGD near minima, revealing nonlinear dynamics that differ from linear analysis.

Paper develops a new local convexity condition for non-isolated minima in non-convex optimization.

Gradient descent with large momentum finds flatter minima.

Noise in RNNs promotes flatter minima and more stable dynamics.

New results show flat minima in neural networks suffer from high dimensionality.

SGD noise helps select flat minima by concentrating in sharp directions and being proportional to loss value.

Study reveals sharp characterisation of local minima in neural network loss landscapes.

New theory shows large learning rates prevent overfitting in neural networks.

GD converges faster to flatter minima than gradient flow in shallow networks.

We establish novel generalization bounds for learning algorithms that converge to global minima. We do so by deriving black-box stability results that only depend on the convergence of a learning algorithm and the geometry around the minimizers of the loss function. The results are shown for nonconvex loss functions sa…

SAM optimizer benefits from normalization, stabilizing and guiding optimization.

Studied how SGD's stability regularization affects generalization in neural networks.

Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous fashion, where delay occurs, can make training even more scalable. However, asynchronous training has its pitfalls, mainly a degradation in…

Study on the noise in SGD minibatches near local minima.

Study examines how training regime affects neural networks' forgetting.

We investigate stability and local minimizing properties of the Riemannian functional defined by the L^p norm of the curvature tensor on the space of Riemannian metrics on a closed manifold. Riemannian metrics with constant curvature and products of such metrics are critical points of this functional. We prove that the…

AdamNX improves Adam's stability by adjusting its learning rate.

New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.

Nonnegative low-rank matrix recovery can have spurious local minima.

Training deep belief networks (DBNs) requires optimizing a non-convex function with an extremely large number of parameters. Naturally, existing gradient descent (GD) based methods are prone to arbitrarily poor local minima. In this paper, we rigorously show that such local minima can be avoided (upto an approximation …

Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively…

Complex-valued neural networks avoid spurious local minima.

We consider the Riemannian functional defined on the space of Riemannian metrics with unit volume on a closed smooth manifold M given by $R_p(g) :=\int_M|R(g)|^pdvg$ where $R(g)$, $dv_g$ denote the corresponding Riemannian curvature, volume form and p is a real number greater than or equal to 2. We prove that $R_p$ res…

We study the stability and convergence of training deep ResNets with gradient descent. Specifically, we show that the parametric branch in the residual block should be scaled down by a factor $τ=O(1/\sqrt{L})$ to guarantee stable forward/backward process, where $L$ is the number of residual blocks. Moreover, we establi…

Enhances deep learning by boosting generalization and convergence.

Paper finds wide minima are better for generalization and proposes a new learning rate schedule.

Truncated SGD with heavy-tailed noise eliminates sharp local minima.

Optimizers find approximate global minima in non-convex problems.

New method shows how order of gradient updates impacts stability and convergence in deep learning.

The study analyzes local minima in ReLU networks and finds low probability of bad local minima.

Physics analogies explain machine learning overfitting control.

In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatl…

Paper proposes faster method to find local minima in nonconvex optimization.

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Global minima found for multidimensional scaling with penalties.

Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss o…

Proposes NRS to find flat minima in deep neural networks.

Piecewise linear activations create many spurious local minima in neural networks.

SGD can jump from high rank minima to low rank minima in DLNs, but not back.

The notion of flat minima has played a key role in the generalization studies of deep learning models. However, existing definitions of the flatness are known to be sensitive to the rescaling of parameters. The issue suggests that the previous definitions of the flatness might not be a good measure of generalization, b…

The stability of the 3-dimensional Hopf vector field, as a harmonic section of the unit tangent bundle, is viewed from a number of different angles. The spectrum of the vertical Jacobi operator is computed, and compared with that of the Jacobi operator of the identity map on the 3-sphere. The variational behaviour of t…

It is well known that (stochastic) gradient descent has an implicit bias towards flat minima. In deep neural network training, this mechanism serves to screen out minima. However, the precise effect that this has on the trained network is not yet fully understood. In this paper, we characterize the flat minima in linea…

New insights into hidden minima in neural networks.

In "Width complexes for knots and 3-manifolds," Jennifer Schultens defines the width complex for a knot in order to understand the different positions a knot can occupy in the 3-sphere and the isotopies between these positions. She poses several questions about these width complexes; in particular, she asks whether the…

Study of SGD with state-dependent noise, improving escape from local minima.

We continue the comparison between lines of minima and Teichmueller geodesics begun in [CRS1]. We show that in the Teichmueller space of a surface S, lines of minima are quasi-geodesic with respect to the Teichmueller metric. The quasi-geodesic constants depend only on the topological type of S.

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). Our analysis relies on the p…

For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We give the locations and losses of differentiable local minima, and show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activati…