arXiv research

Complex-valued neural networks avoid spurious local minima.

Deep linear networks avoid spurious local minima under certain conditions.

Paper proves conditions for nonconvex matrix recovery to avoid spurious local minima.

The paper proves deep ReLU networks avoid spurious local minima in NTK regime.

When the linear measurements of an instance of low-rank matrix recovery satisfy a restricted isometry property (RIP)---i.e. they are approximately norm-preserving---the problem is known to contain no spurious local minima, so exact recovery is guaranteed. In this paper, we show that moderate RIP is not enough to elimin…

Piecewise linear activations create many spurious local minima in neural networks.

New insights into spurious local minima in k-means clustering.

We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insigh…

We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto \sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the squared loss. We provide a computer-assisted proof that even if the input distribution is standard Gaussian, even if the dime…

The paper studies quadratic neural networks, proving existence of spurious minima and saddle points.

The paper analyzes how good initial guesses affect the amount of data needed for low-rank matrix recovery.

We consider the non-square matrix sensing problem, under restricted isometry property (RIP) assumptions. We focus on the non-convex formulation, where any rank-$r$ matrix $X \in \mathbb{R}^{m \times n}$ is represented as $UV^\top$, where $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{n \times r}$. In this paper…

Nonnegative low-rank matrix recovery can have spurious local minima.

We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a global optimum. Together with a curvature bound at saddle points, this yields a polynomial ti…

Gradient flow in phase retrieval escapes spurious minima with high probability.

DMs emerge from DenseAMs, transitioning from memorization to generalization.

Study on descent algorithms in spiked matrix-tensor models, revealing performance issues and solutions.

Gradient descent and SGD can converge to max-margin directions in ReLU models.

Study reveals sharp characterisation of local minima in neural network loss landscapes.

Matrix completion is a basic machine learning problem that has wide applications, especially in collaborative filtering and recommender systems. Simple non-convex optimization algorithms are popular and effective in practice. Despite recent progress in proving various non-convex algorithms converge from a good initial …

Gradient-flow helps find good minima in complex models.

New framework for learning KR maps from data, ensuring stable generalization.

Convex clustering solves a stable optimization problem for clustering.

We consider the problem of learning a one-hidden-layer neural network with non-overlapping convolutional layer and ReLU activation, i.e., $f(\mathbf{Z}, \mathbf{w}, \mathbf{a}) = \sum_j a_jσ(\mathbf{w}^T\mathbf{Z}_j)$, in which both the convolutional weights $\mathbf{w}$ and the output weights $\mathbf{a}$ are paramete…

Least symmetry breaking principle explains SGD's local minima in shallow ReLU networks.

Self-training avoids spurious features in domain adaptation.

Paper analyzes noisy low-rank matrix optimization, improving RIP bounds and convergence rates.

In this paper we develop a new framework that captures the common landscape underlying the common non-convex low-rank matrix problems including matrix sensing, matrix completion and robust PCA. In particular, we show for all above problems (including asymmetric cases): 1) all local minima are also globally optimal; 2) …

Population risk is always of primary interest in machine learning; however, learning algorithms only have access to the empirical risk. Even for applications with nonconvex nonsmooth losses (such as modern deep networks), the population risk is generally significantly more well-behaved from an optimization point of vie…

Geometric study of linear neural networks identifies pure and spurious critical points.

Paper shows no spurious local minima in a specific matrix factorization problem.

Analyzes the Hessian of ReLU networks, proving skewed eigenvalue distribution.

SGD with over-param. makes neural net landscape connected, aiding optimization.

This work studies low-rank approximation of a positive semidefinite matrix from partial entries via nonconvex optimization. We characterized how well local-minimum based low-rank factorization approximates a fixed positive semidefinite matrix without any assumptions on the rank-matching, the condition number or eigensp…

KSD Descent uses KSD to sample from a target distribution efficiently.

We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one ne…

Sharp global guarantees for noisy overparameterized low-rank recovery.

Overparametrization improves QNN trainability by reducing spurious local minima.

This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.

New method smooths optimization for sparse regularization.

Study reveals different types of critical points in shallow neural networks.

This work shows that a simple local search can recover true principal components in non-negative rank-1 RPCA.

Rapid mixing of Langevin dynamics on Riemannian manifolds

We describe loss surfaces using topological Betti numbers.

Removes all bad local minima without adding extra units.

Study reveals properties of local minima in ReLU networks.

This paper proves adding neurons eliminates all bad local minima in deep learning.

Unified approach to characterize and regularize deep neural network local minima.