Removes all bad local minima without adding extra units.
problem Neural network loss landscapes have bad local minima.
method Converts all bad local minima to non-local minima at infinity using auxiliary parameters.
result Removes all bad local minima from any loss landscape.
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatl…
The study analyzes local minima in ReLU networks and finds low probability of bad local minima.
problem Understanding the existence and probability of local minima in ReLU networks.
method Theoretical analysis combined with linear programming and experiments on MNIST and CIFAR-10 datasets.
result No bad differentiable local minima found almost everywhere in weight space.
Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection…
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are gu…
This paper proves adding neurons eliminates all bad local minima in deep learning.
problem Eliminating all suboptimal local minima in deep learning models.
method Adding one special neuron per output unit eliminates all suboptimal local minima of deep neural networks.
result At every local minimum, the original neural network parameters are a global minimum.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
New technique improves imitation learning by preventing local minima and exploring states.
problem Behavioral cloning gets stuck in local minima and lacks effective exploration.
method Two-phase model with sampling mechanisms and self-attention modules.
result Significantly outperforms previous state-of-the-art in various environments.
We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insigh…
The paper shows deep neural networks have no bad local minima and no diverging paths to infinity.
problem The risk of diverging to infinity in deep neural networks.
method Mathematical analysis of regularizers and loss functions.
result For a large class of over-parameterized deep neural networks, the loss function has no bad local minima and no decreasing paths to infinity.
MCN improves deep neural networks by bettering local minima and generalizing well.
problem Bad local minima and poor generalization in deep neural networks.
method Introducing Maximum-and-Concatenation Networks (MCN) to eliminate bad local minima and improve generalization.
result MCN can autonomously improve local minima's goodness by increasing network depth.
There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that tra…
Wide networks avoid bad local minima, proving phase transition.
problem Understanding the optimization landscape of neural networks.
method Proving phase transitions in loss surfaces of wide and narrow networks.
result Phase transition from narrow to wide networks: no bad local minima.
We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…
Deep learning networks have connected sublevel sets, avoiding local minima.
problem Finding local minima in deep learning networks.
method Analyzing sublevel sets of loss functions in over-parameterized neural nets.
result Sublevel sets are connected and unbounded, ensuring all global minima are accessible.
Analyzes minima of deep linear networks with weight decay.
problem Understanding the loss landscape of deep neural networks.
method Analytical solutions for global minima with weight decay and stochastic neurons.
result The origin is a special point with qualitatively different minima in networks with more than 1 hidden layer.
This paper compares EM and GD in two-component mixture models, finding EM escapes bad local optima more reliably.
problem Understanding the convergence of EM and GD in mixture models, especially in regions where one component is missing.
method Analyzing regions called one-cluster regions in two-component mixture models of Gaussians and Bernoullis, comparing the propensity of EM and GD to converge to these regions.
result EM escapes one-cluster regions exponentially fast, while GD escapes them linearly fast, indicating EM is less likely to converge to bad local optima.
Gradient-based methods find saddle points, not critical points, in neural networks.
problem Gradient-based optimization methods converge to saddle points rather than critical points in deep neural networks.
method Critical point-finding methods used to analyze neural network losses.
result Gradient-based methods often converge to or pass through gradient-flat regions, where gradient norm has a stationary point.
SGD can converge to bad models if not explicitly regularized.
problem Overparameterized neural networks can converge to poorly-generalizing models.
method Train with random labels first, then correct labels; observe SGD behavior.
result SGD can reach bad global minima if not explicitly regularized.
The paper tackles how weakly interacting agents can learn to escape flat minima in machine learning.
problem Understanding how flat minima affect generalization in machine learning.
method Formalizes the problem as an optimization problem with weakly interacting agents, reviews stochastic processes, and proposes a new algorithm.
result Illustrates the potential of an algorithmic framework for escaping flat minima.
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015. With no unrealistic assumption, we first prove the following statements for the squared loss function of deep linear neural networks with any depth and any widths…
Gradient-flow helps find good minima in complex models.
problem Understanding why gradient-based algorithms work in non-convex optimization.
method Kac-Rice analysis and gradient-flow from statistical physics.
result Gradient-flow finds good global minima in the presence of many spurious local minima.
This paper explores optimization methods for deep learning.
problem How to effectively train neural networks.
method Discusses gradient explosion/vanishing, initialization, normalization, SGD, adaptive gradient methods, distributed methods, and global issues.
result Provides theoretical insights and practical solutions for neural network training.
New proof shows deep neural nets can have sub-optimal local minima.
problem Can over-parameterization eliminate sub-optimal local minima in deep neural networks?
method Counter-example with generic input data and non-linear activation functions.
result Sub-optimal local minima exist in deep neural networks regardless of width.
We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …
New research shows posterior collapse in VAEs isn't just about KL-divergence.
problem Posterior collapse in Variational Autoencoders (VAEs).
method Analyzes the loss surface of deep autoencoder networks and proves the existence of bad local minima.
result Posterior collapse in VAEs is caused by bad local minima, not just KL-divergence.
Geometric study of linear neural networks identifies pure and spurious critical points.
problem Understanding the landscape of loss functions in linear neural networks.
method Geometric properties of functional spaces and parameterization analysis.
result Different phenomena cause the absence of bad local minima in linear networks, depending on the architecture and loss function.
We analyze the loss landscape and expressiveness of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a "wide" layer which has more neurons than the number of training samples. This condition holds e.g. for the…
Transformers with multiple layers learn to estimate bigram distributions, while single-layer models often get stuck in unigram local minima.
problem Understanding the sequential modeling capabilities of transformers using Markov chains.
method Introducing a new framework to analyze transformers via Markov chains, characterizing their loss landscapes.
result Single-layer transformers often get stuck in local minima representing the unigram distribution, while deeper models reliably converge to the ground-truth bigram.
Our derivation of the distribution function for future returns is based on the risk neutral approach which gives a functional dependence for the European call (put) option price, C(K), given the strike price, K, and the distribution function of the returns. We derive this distribution function using for C(K) a Black-Sc…
SSRGD finds local minima in nonconvex problems with simple gradient updates.
problem Finding local minima in nonconvex optimization problems.
method Simple perturbed stochastic recursive gradient descent (SSRGD).
result SSRGD finds (ε,δ)-second-order stationary points efficiently. The counting grid is a grid of microtopics, sparse word/feature distributions. The generative model associated with the grid does not use these microtopics individually. Rather, it groups them in overlapping rectangular windows and uses these grouped microtopics as either mixture or admixture components. This paper bui…
Minimizing non-convex and high-dimensional objective functions is challenging, especially when training modern deep neural networks. In this paper, a novel approach is proposed which divides the training process into two consecutive phases to obtain better generalization performance: Bayesian sampling and stochastic op…
Proposes a new factor to improve BAB strategies by recognizing bad-beta assets.
problem Investors often misprice assets based on beta, ignoring bad-beta.
method Double-sorting on beta and bad-beta to create a new factor.
result The Betting Against Bad Beta factor improves BAB strategies.
The abstract constructs a set of bad 3-orbifolds and shows how any bad 3-orbifold can be transformed into a good one.
problem Characterizing and transforming bad 3-orbifolds into good ones.
method Explicit construction of bad 3-orbifolds and a method of cutting-and-capping to transform them.
result Any bad 3-orbifold can be transformed into a good 3-orbifold through a finite number of operations.
Mixed linear regression involves the recovery of two (or more) unknown vectors from unlabeled linear measurements; that is, where each sample comes from exactly one of the vectors, but we do not know which one. It is a classic problem, and the natural and empirically most popular approach to its solution has been the E…
Matrix completion is a well-studied problem with many machine learning applications. In practice, the problem is often solved by non-convex optimization algorithms. However, the current theoretical analysis for non-convex algorithms relies heavily on the assumption that every entry is observed with exactly the same pro…
Gradient descent fails to train two-layer ReLU networks, leading to poor performance.
problem Gradient descent training of two-layer ReLU networks initialized by He et al. (2015) fails to find optimal solutions.
method Gradient descent on a least-squares loss for training two-layer (Leaky)ReLU networks.
result Gradient descent only finds bad local minima, leading to linear regression for non-linear target functions.
New attacks reduce bad queries in black-box classifiers, improving effectiveness.
problem Evasion attacks often issue many bad queries, which are costly.
method Design new attacks that minimize bad queries while increasing non-bad queries.
result Reduces the number of bad queries by 1.5-7.3 times, improving attack effectiveness.
SAMformer improves transformer performance in time series forecasting.
problem Transformers struggle with multivariate long-term forecasting.
method Sharpness-aware minimization and channel-wise attention.
result SAMformer surpasses state-of-the-art methods in multivariate time series forecasting.
Unified-GAN improves SSL with both good and bad samples.
problem Improve semi-supervised learning performance with limited labeled data.
method Unified-GAN combines adversarial training with good and bad samples.
result Unified-GAN achieves state-of-the-art performance and robustness to varying labeled data.
SGD favors flat minima exponentially more than sharp minima in deep learning.
problem Understanding how SGD selects flat minima in deep learning.
method Developed a density diffusion theory (DDT) to analyze minima selection.
result SGD exponentially favors flat minima over sharp minima due to Hessian-dependent noise.
RMCSE improves voltage estimation in low-observability distribution systems.
problem Insufficient measurements in distribution system state estimation.
method Combines matrix completion and power system model, minimizes rank and residual with different weights.
result Robust voltage estimation in low-observability systems without bad data detection.
Study finds stocks with common firm fears earn lower returns.
problem Identifying and quantifying firm-level investor fears.
method Analysis of equity options to identify common firm-level fears and their impact on stock returns.
result Stocks with exposure to common bad fears earn lower returns and require higher compensation.
This paper introduces a gradient analysis framework to improve language model performance by rewarding good examples and penalizing bad ones.
problem Improving language model output quality by penalizing bad examples.
method Gradient analysis of loss functions to reward good examples and penalize bad ones.
result ExMATE is superior to MLE and combining DPO with ExMATE enhances performance.
Improves VAT method for semi-supervised learning with simpler techniques.
problem Improving semi-supervised learning performance with robust classifiers.
method Adopts Bad GAN approach to generate bad samples and employs adversarial training.
result Achieves competitive performance with fewer computations.