New framework for learning KR maps from data, ensuring stable generalization.
problem Learning monotone triangular transport maps efficiently and accurately.
method General framework using invertible transformations of smooth functions, ensuring no spurious local minima.
result Unique global minimizer corresponds to the KR map under certain conditions.
New theory shows large learning rates prevent overfitting in neural networks.
problem Generalization of two-layer ReLU neural networks in noisy regression problems.
method Gradient descent with constant learning rate converges to stable minima.
result Gradient descent with large learning rates finds smooth, sparse fits.
Convex clustering solves a stable optimization problem for clustering.
problem Clustering with stable and scalable solutions.
method Solving a convex optimization problem with a single tuning parameter.
result The optimization problem has a unique global minimizer stable to inputs.
New bounds show learning algorithms stable near global minima.
problem Generalization of learning algorithms converging to global minima.
method Black-box stability results based on convergence and loss function geometry.
result Stability of optimization algorithms like SGD, GD, RCD, and SVRG.
New results show flat minima in neural networks suffer from high dimensionality.
problem Flat minima in neural networks generalize poorly in high dimensions.
method Theoretical analysis of two-layer ReLU networks with multivariate inputs.
result Flat minima lead to exponentially slower convergence in high dimensions.
Study nonconvex matrix completion for low-rank approximation without rank assumptions.
problem Low-rank approximation of positive semidefinite matrices from partial entries.
method Nonconvex optimization, local-minimum analysis, no spurious local minima.
result Improved sampling rate for nonconvex matrix completion with no spurious local minima.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Deep linear networks with arbitrary loss have all local minima as global minima.
problem Finding optimal solutions in deep linear networks with arbitrary convex losses.
method Provided a short and elementary proof for all local minima being global minima under specific conditions.
result All local minima are global minima for deep linear networks with certain layer widths.
Removes all bad local minima without adding extra units.
problem Neural network loss landscapes have bad local minima.
method Converts all bad local minima to non-local minima at infinity using auxiliary parameters.
result Removes all bad local minima from any loss landscape.
Study reveals properties of local minima in ReLU networks.
problem Understanding the loss landscape of neural networks.
method Theoretical analysis of one-hidden-layer ReLU networks.
result All differentiable local minima are global within certain regions.
This work examines the stability of GD and SGD near minima, revealing nonlinear dynamics that differ from linear analysis.
problem The stability of optimization algorithms like GD and SGD near minima is not well understood.
method The authors derive an exact criterion for stable oscillations of GD near minima in the multivariate setting, considering high-order derivatives.
result Nonlinear dynamics can diverge in expectation even if a single batch is unstable, challenging linear analysis.
This paper proves adding neurons eliminates all bad local minima in deep learning.
problem Eliminating all suboptimal local minima in deep learning models.
method Adding one special neuron per output unit eliminates all suboptimal local minima of deep neural networks.
result At every local minimum, the original neural network parameters are a global minimum.
Unified approach to characterize and regularize deep neural network local minima.
problem Characterize and improve generalizability of deep neural network local minima.
method Information-theoretic Fisher information metric for local minima characterization and regularization.
result Unified approach successfully characterizes and improves generalizability of DNNs.
The study examines stable regions in weighted manifolds with boundary properties.
problem Studying stable regions in weighted manifolds with boundary properties.
method Using deformations constructed from parallel vector fields tangent to the boundary, the study deduces rigidity properties for stable sets.
result The classification of stable sets in some Riemannian cylinders and uniqueness results for minimizers.
Adding one neuron fixes neural network's bad local minima.
problem Non-convexity of loss function leading to many bad local minima in neural networks.
method Adding one special neuron with a skip connection to the output or one per layer.
result Every local minimum becomes a global minimum after adding one neuron.
Small non-linearities in activation functions lead to bad local minima in neural networks.
problem Bad local minima in neural networks due to small non-linearities.
method Proved spurious local minima for ReLU(-like) networks and constructed counterexamples for more general activations.
result For almost all practical datasets, there exist infinitely many local minima in ReLU(-like) networks.
The study analyzes local minima in ReLU networks and finds low probability of bad local minima.
problem Understanding the existence and probability of local minima in ReLU networks.
method Theoretical analysis combined with linear programming and experiments on MNIST and CIFAR-10 datasets.
result No bad differentiable local minima found almost everywhere in weight space.
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatl…
New analysis shows SGD prefers wide minima due to heavy-tailed noise.
problem Analyzing the behavior of SGD in deep learning settings.
method Invoking the generalized CLT to model SGD as driven by a Lévy motion.
result Established a connection between SGD convergence and tail-index of noise.
Minimal Gaussian surface area is achieved by cones over a regular simplex for m>3 sets partitioning Rn.
problem Finding the minimal Gaussian surface area of m sets partitioning Rn. method Volume-preserving variations of the sets, avoiding matrix-valued partial differential inequalities.
result Strengthened Milman-Neeman Gaussian multi bubble theorem and first known dimension-independent bounds for the Plurality is Stablest Conjecture.
Optimizers find approximate global minima in non-convex problems.
problem Understanding why local methods solve non-convex optimization problems.
method Formalizing the hypothesis that many local minima are approximately global minima.
result Most local minima of practical non-convex objectives are approximately global minima.
Wide neural networks have non-attracting local minima.
problem Understanding the impact of suboptimal local minima in deep and wide neural networks.
method Construction of non-attracting local minima and saddle points in wide neural networks.
result Wide neural networks have non-attracting local minima, even though they are not negatively impacted by suboptimal local optima.
Simple neural networks can have many deceptive local minima.
problem Training simple ReLU neural networks can lead to spurious local minima.
method Computer-assisted proof and concentration of measure argument.
result Even in high dimensions, spurious local minima are common for ReLU networks.
Truncated SGD with heavy-tailed noise eliminates sharp local minima.
problem Avoiding sharp local minima in deep learning models.
method Truncated SGD with heavy-tailed gradient noise.
result Truncated SGD can eliminate sharp local minima entirely from its training trajectory.
ResNets with depth and nonlinearity avoid bad local minima.
problem Avoiding bad local minima in deep learning models.
method Proving depth and nonlinearity in ResNets create no bad local minima.
result ResNets with depth and nonlinearity have values no worse than global minimum and can improve further.
Paper proves conditions for nonconvex matrix recovery to avoid spurious local minima.
problem Ensuring no spurious local minima in nonconvex matrix recovery.
method Sharp restricted isometry bounds proof technique.
result RIP constant of δ < 1/2 is necessary and sufficient for exact recovery.
The paper examines how depth and width affect local minima in neural networks.
problem Understanding the impact of depth and width on local minima in neural networks.
method Theoretical analysis and empirical validation of deep nonlinear neural networks with squared loss.
result The quality of local minima improves towards global minimum values as depth and width increase.
In "Width complexes for knots and 3-manifolds," Jennifer Schultens defines the width complex for a knot in order to understand the different positions a knot can occupy in the 3-sphere and the isotopies between these positions. She poses several questions about these width complexes; in particular, she asks whether the…
Paper develops a new local convexity condition for non-isolated minima in non-convex optimization.
problem Lack of theory for non-isolated minima in non-convex optimization.
method Formulates a new local convexity condition and studies SGD convergence under this condition.
result Shows SGD can converge locally under the new condition.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Reduces non-convex optimization to finding local minima using gradients.
problem Finding local minima in non-convex optimization problems.
method Reduces non-convex optimization to gradient-based methods.
result Turns various optimization algorithms into local minimum finding methods.
The paper proves deep ReLU networks avoid spurious local minima in NTK regime.
problem The existence of spurious local minima in deep ReLU neural networks.
method Theoretical proof under Neural Tangent Kernel regime.
result Deep ReLU networks do not lie in spurious local minima in NTK regime.
Solves local minima problems on smooth manifolds.
problem Local minima issues on smooth manifolds.
method Introducing valley functions and applying Morse's lemma.
result Eliminates critical points and reduces to 1D.
Deep linear networks avoid spurious local minima under certain conditions.
problem Existence of spurious local minima in deep linear networks.
method Reduction to two-layer case, quadratic loss analysis, and perturbation argument to show full rank property.
result Deep linear networks have no spurious local minima under specific conditions.
The paper tackles spurious local minima in empirical risk and proposes an SGD-based algorithm to find ε-approximate local minima of the underlying function.
problem Spurious local minima in empirical risk for nonconvex nonsmooth losses.
method A simple SGD-based algorithm on a smoothed version of the empirical risk function.
result The algorithm finds ε-approximate local minima of the underlying function F while avoiding shallow local minima arising from the tolerance ν. Paper shows moderate RIP is insufficient for avoiding spurious local minima in matrix recovery.
problem The need for moderate RIP to avoid spurious local minima in matrix recovery.
method Analyzes the necessity of RIP constants and provides counterexamples.
result Counterexamples show spurious local minima exist even with moderate RIP.
Study of SGD with state-dependent noise, improving escape from local minima.
problem Understanding and improving the dynamics of SGD in non-convex optimization.
method Formal study on SGD with state-dependent noise, proposing power-law dynamic with state-dependent diffusion.
result Power-law dynamic can escape from sharp minima exponentially faster than flat minima.
Stochastic NGD approximates Bayesian posterior samples near local minima.
problem Approximating Bayesian uncertainty in model parameters near local minima.
method Develops minibatch natural gradient descent (NGD) and introduces stochastic NGD to preserve Bayesian properties.
result Minibatch NGD's stationary distribution approaches a Bayesian posterior near local minima with small learning rates.
Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
We investigate stability and local minimizing properties of the Riemannian functional defined by the L^p norm of the curvature tensor on the space of Riemannian metrics on a closed manifold. Riemannian metrics with constant curvature and products of such metrics are critical points of this functional. We prove that the…
This paper compares EM and GD in two-component mixture models, finding EM escapes bad local optima more reliably.
problem Understanding the convergence of EM and GD in mixture models, especially in regions where one component is missing.
method Analyzing regions called one-cluster regions in two-component mixture models of Gaussians and Bernoullis, comparing the propensity of EM and GD to converge to these regions.
result EM escapes one-cluster regions exponentially fast, while GD escapes them linearly fast, indicating EM is less likely to converge to bad local optima.
MCN improves deep neural networks by bettering local minima and generalizing well.
problem Bad local minima and poor generalization in deep neural networks.
method Introducing Maximum-and-Concatenation Networks (MCN) to eliminate bad local minima and improve generalization.
result MCN can autonomously improve local minima's goodness by increasing network depth.
SGD noise helps select flat minima by concentrating in sharp directions and being proportional to loss value.
problem Understanding the implicit regularization of SGD and selecting flat minima in over-parameterized models.
method Relating SGD's linear stability to the Frobenius norm of the Hessian and analyzing the alignment property of SGD noise.
result Flat minima are linearly stable for SGD, and their sharpness is bounded independently of model size and sample size.
New algorithm finds local minima faster than SGD for nonconvex functions.
problem Finding local minima in nonconvex optimization efficiently.
method Stochastic cubic regularization of Newton method.
result Matches best-known result for local minima without acceleration.
We use barcodes to analyze neural networks' loss surfaces, revealing important properties.
problem Understanding the topology of neural networks' loss surfaces.
method Topological data analysis using Morse complexes and barcodes.
result Barcodes of local minima are located in a small part of the loss function's range and decrease with network depth and width.
New algorithm trains neural networks faster than SGD.
problem Training neural networks to local minima efficiently.
method Stochastic algorithm with O(ε−3.25) backpropagations. result Trains to ε-approximate local minima faster than SGD.