New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
Study landscape of non-convex empirical risk with degenerate population risk.
problem Degenerate non-convex population risk in machine learning problems.
method Analyze population risk first, then connect to empirical risk landscape.
result Established correspondence between empirical and population risk critical points.
The paper analyzes phase retrieval under limited samples, ensuring a benign local landscape for convergence.
problem Ensuring a benign local landscape for phase retrieval under limited samples.
method Fine-grained analysis of local landscape properties under the regime of limited samples.
result Gradient descent can converge to an o d ( 1 ) o_d(1) o d ( 1 ) -loss solution exponentially fast under certain conditions. Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
We solve the optimization of two-layer ReLU networks using convex math.
problem Optimizing two-layer ReLU neural networks.
method Exact characterization of optimal solutions via convex optimization.
result We prove that all globally optimal solutions can be found via convex optimization.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
New framework shows all local minima are globally optimal in non-convex low-rank problems.
problem Non-convex low-rank problems, including matrix sensing, completion, and robust PCA.
method Developed a new framework to analyze the optimization landscapes of these problems.
result All local minima are also globally optimal and no high-order saddle points exist.
Exploring loss landscapes of XOR networks reveals complex structures.
problem Understanding the optimization landscape of XOR networks.
method Using molecular science optimization tools, analyzing the number and types of stationary points.
result The landscape of XOR networks becomes more convex with increased regularisation, embedding smaller networks in larger ones.
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
The paper analyzes adaptive algorithms in non-convex optimization landscapes.
problem Analyzing adaptive algorithms in non-convex optimization landscapes.
method Stochastic algorithms with decreasing step-size, considering mini-batches and noise.
result Established almost sure convergence to critical points and minimizers.
The paper studies the landscape of empirical risk for non-convex losses.
problem Understanding the complexity of non-convex losses in high-dimensional estimation.
method Analyzing the landscape of the empirical risk, focusing on stationary points and their properties.
result Uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, ensuring good properties of the population risk can be carried to the empirical risk.
Almost all local minima in neural networks are strongly convex.
problem The prevalence of strongly convex neighborhoods around local minima in neural network optimization landscapes.
method Rigorous analysis of shallow neural networks with analytic activation functions, dividing parameter space into efficient and redundant domains.
result For shallow neural networks on the efficient domain, almost all local minima are strongly convex.
Neural networks' energy landscape is surprisingly flat, suggesting minimal structural changes between minima.
problem Understanding the structure of neural network energy landscapes.
method Constructing continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100.
result Paths between minima are essentially flat in both training and test landscapes, implying minimal structural changes.
Study on negative eigenvalues in deep neural networks' loss landscapes.
problem Understanding the non-convex nature of deep neural networks' loss functions.
method Examined the Hessian matrix's eigendecompositions to analyze negative eigenvalues.
result Negative eigenvalues are crucial for understanding the loss landscape of deep networks.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Quantum annealing outperforms classical in solving non-convex optimization problems crucial for machine learning.
problem Solving non-convex optimization problems efficiently.
method Designing a classical energy function and adding a quantum transverse field to facilitate tunneling.
result Quantum annealing converges efficiently to optimal solutions in a wide class of non-convex problems, unlike classical thermal annealing.
This paper explores neural network loss landscapes and their effects on generalization.
problem Understanding the structure of neural network loss functions and their impact on generalization.
method Simple filter normalization and various visualization methods to explore loss landscape structure and network architecture effects.
result Visualizations reveal how network architecture and training parameters affect loss landscape curvature and minimizers.
We analyze a non-convex landscape for robust subspace recovery and prove exact recovery conditions.
problem Analyzing the robustness of subspace recovery in non-convex energy landscapes.
method Mathematical analysis and proof of conditions for exact recovery of the underlying subspace.
result A geodesic gradient descent method can exactly recover the underlying subspace under specific conditions.
Optimizes MMD learning for generative models with theoretical guarantees.
problem Theoretical guarantees for optimizing non-convex MMD objectives.
method Analyzes MMD optimization landscape for specific distributions.
result Gradient-based methods globally minimize MMD objective for certain distributions.
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Monotonic Linear Interpolation property in neural networks persists despite non-convexity.
problem Understanding the geometric properties of neural network loss landscapes.
method Tools from differential geometry to analyze the monotonicity of neural network weights.
result Sufficient conditions for the Monotonic Linear Interpolation property under mean squared error.
In many statistical learning problems, the target functions to be optimized are highly non-convex in various model spaces and thus are difficult to analyze. In this paper, we compute \emph{Energy Landscape Maps} (ELMs) which characterize and visualize an energy function with a tree structure, in which each leaf node re…
Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.
problem Understanding the impact of over-parameterization on optimization landscapes of shallow neural networks.
method Analyzing a simple ReLU neural network with Gaussian inputs, focusing on optimization properties and landscape changes.
result Over-parameterization makes the objective function one-point strongly convex in most directions, aiding optimization.
Averaged Gradient Descent improves performance in rough landscapes.
problem Minimizing strongly non-convex functions in high-dimensional estimation problems.
method Empirical average of gradients over random positions in parameter space.
result Averaged Gradient Descent outperforms existing methods in tensor PCA.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Designs a non-convex objective function to learn one-hidden-layer neural networks.
problem Learning one-hidden-layer neural networks with Gaussian input and nonnegative label.
method Analytic formula for population risk, landscape design of non-convex objective function G ( ⋅ ) G(\cdot) G ( ⋅ ) , stochastic gradient descent. result Stochastic gradient descent on G G G converges to the global minimum and learns the ground-truth parameters. Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
problem Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
method Identifying entropic barriers arising from the interplay between curvature variations along low-loss paths and noise in optimization dynamics.
result Curvature-induced entropic forces bias noisy dynamics back toward the endpoints, explaining the confinement and connectivity of solutions.
The paper proves skip connections help neural networks avoid shallow local minima.
problem Understanding how skip connections affect the loss landscape of deep neural networks.
method Theoretical analysis of the topology of loss landscapes of deep ReLU neural networks with skip connections.
result Skip connections help control the connectedness of sub-level sets, avoiding shallow local minima.
New function class characterizes loss landscape of deep neural networks without over-parametrization.
problem Complex loss landscape of deep neural networks without over-parametrization.
method Proposed a novel class of functions to characterize loss landscape without over-parametrization.
result Gradient-based optimizers possess theoretical guarantees of convergence under the new function class assumption.
This work analyzes a two-stage algorithm for single index models, showing precise asymptotics of gradient descent.
problem Learning single index models with non-convex optimization.
method Spectral initialization followed by gradient descent, with detailed analysis of dynamics and asymptotics.
result Gradient descent converges to long-time fixed points in the large system limit, representing mean field behavior.
Geometric study of linear neural networks identifies pure and spurious critical points.
problem Understanding the landscape of loss functions in linear neural networks.
method Geometric properties of functional spaces and parameterization analysis.
result Different phenomena cause the absence of bad local minima in linear networks, depending on the architecture and loss function.
This work explores the non-convex optimization in compressive learning and the performance of heuristics.
problem The challenge of learning from compressed representations in compressive learning.
method Numerical simulations of the non-convex optimization landscape and heuristic performance.
result Properties of the non-convex optimization landscape and heuristic performance are explored.
We found a 'Goldilocks zone' in neural network loss landscapes that correlates with good initialization.
problem Understanding and optimizing neural network loss landscapes for better initialization.
method Random and low-dimensional hypersurfaces to evaluate the Hessian of loss functions.
result The Goldilocks zone is a region of unusually high convexity and positive curvature, correlated with good network performance.
Study of neural network training using mean-field Langevin dynamics and energy landscapes.
problem Understanding convergence of stochastic gradient algorithms for non-convex learning tasks.
method Leverage infinite-dimensional convexification, Mean-Field Langevin Dynamics, and energy functional.
result Gradient flow converges to a unique minimizer in 2-Wasserstein metric, with exponential convergence under regularisation.
This paper analyzes the landscape of supervised contrastive loss in over-parameterized networks.
problem Understanding the structure of solutions in over-parameterized networks under supervised contrastive loss.
method Analytical approach using unconstrained features model (UFM) to study the solutions of SC loss minimization.
result All local minima of SC loss are global minima in over-parameterized networks, and the minimizer is unique (up to rotation).
Study shows overparametrization can shift and bend loss landscapes, affecting signal recovery.
problem Understanding how overparametrization affects loss landscapes in neural networks.
method Field theory analysis of Hessian spectrum at initialization.
result Overparametrization can shift the BBP transition point, potentially reaching weak-recovery threshold.
This work refines claims about neural network connectivity, showing that simultaneous linear connectivity is possible under certain conditions.
problem Neural networks' loss landscapes are non-convex due to permutation symmetries, leading to high loss barriers between permuted networks.
method The authors introduce and analyze three claims of increasing strength regarding the connectivity of neural networks, focusing on permutations that align networks.
result The authors provide evidence that strong linear connectivity may be possible under certain conditions, specifically when interpolating among three networks of increasing width.
Analyzes optimal learning rate schedules in high-dimensional non-convex optimization problems.
problem Optimizing high-dimensional non-convex loss landscapes.
method Langevin optimization with learning rate decaying as \(η(t) = t^{-β}\).
result To speed up optimization without getting stuck in saddles, a decay rate \(β < 1\) is optimal, contrary to convex setups where \(β = 1\).
Deep learning dynamics exhibit anomalous superdiffusion initially, aiding escape from local minima.
problem Understanding the dynamics of learning in deep neural networks.
method Novel analysis of SGD dynamics and loss landscape structure.
result SGD exhibits anomalous superdiffusion initially, transitioning to subdiffusion as learning progresses.
Study on the optimization landscape of half-rectified networks without simplifying assumptions.
problem Understanding the optimization landscape of deep neural networks, focusing on half-rectified networks.
method Theoretical analysis and empirical study of gradient descent on half-rectified networks.
result Proves that half-rectified single layer networks are asymptotically connected and provides bounds on the interplay between data distribution and model over-parametrization.
The paper reveals surprising star-shaped connectivity in neural networks.
problem Understanding mode connectivity in neural network landscapes.
method Fine-grained analysis of connectivity in overparameterized and finite minima cases.
result Star-shaped connectivity exists in neural network landscapes, suggesting near convexity.
Study non-convex matrix factorization using Riemannian geometry.
problem Matrix completion via non-convex optimization.
method Optimization over a Grassmannian manifold, analyzing principal angles.
result Geodesically convex region in matrix completion cost function.
Visualizes optimization landscapes to understand FCN performance.
problem Understanding why FCNs perform well empirically.
method Visualizing objective functions in 3D space, comparing networks, investigating skip-layer connections, and analyzing loss surfaces.
result Skip-layer connections in FCNs promote flat optimization landscapes, leading to better generalization.
Over-parametrized networks with quadratic activations can find globally optimal solutions for convex losses.
problem Finding globally optimal solutions in neural networks with quadratic activations.
method Analyzing the landscape properties of loss functions and using Rademacher complexity for generalization.
result Over-parametrization with k ≥ 2 n k \ge \sqrt{2n} k ≥ 2 n enables finding globally optimal solutions for convex losses. New algorithm learns convolutional neural networks with overlapping patches.
problem Learning convolutional neural networks with overlapping patches.
method Algorithm draws from isotonic regression and landscape analysis.
result Algorithm works for general class of patches, including common computer vision structures.
Paper shows non-convex loss functions can be optimized efficiently.
problem Optimizing non-convex loss functions is challenging.
method Uses stochastic variance reduction methods to find global optimal solutions.
result Stochastic variance reduction methods converge to global optimal with linear rate.
Theory explains symmetry and saddle points in nonconvex optimization landscapes.
problem Understanding the optimization landscape of nonconvex matrix factorization problems.
method Characterizing stationary points and null spaces via invariant groups.
result Identifies infinitely many nonisolated strict saddle points and global minima.