Equivalent formulations for low-rank matrix optimization are proven.
problem Low-rank matrix optimization with rank constraints.
method Established geometric landscape connections between manifold and factorization formulations.
result Equivalence between manifold and factorization formulations at FOSPs, SOSPs, and strict saddles.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
Paper explores SNN for learning spectral geometric info from data.
problem Challenges in applying traditional eigensolvers to big data.
method Introduces Spectral Neural Networks (SNN) as an alternative.
result Investigates tradeoffs and optimization landscape of SNN.
We study rough high-dimensional landscapes in which an increasingly stronger preference for a given configuration emerges. Such energy landscapes arise in glass physics and inference. In particular we focus on random Gaussian functions, and on the spiked-tensor model and generalizations. We thoroughly analyze the stati…
Experimental fractal landscape dynamics observed in emulsions.
problem Understanding anomalous motions in soft glassy materials.
method Quantitative analysis of oil droplet trajectories in dense emulsions.
result Experimental fractal geometry matches computational model of soft glassy dynamics.
In this paper we investigate the properties of series of vacua in the string theory landscape. In particular, we study minima to the flux potential in type IIB compactifications on the mirror quintic. Using geometric transitions, we embed its one dimensional complex structure moduli space in that of another Calabi-Yau …
We study nonconvex optimization landscapes for learning overcomplete representations, including learning (i) sparsely used overcomplete dictionaries and (ii) convolutional dictionaries, where these unsupervised learning problems find many applications in high-dimensional data analysis. Despite the empirical success of …
GGA improves untrustworthy prediction detection in neural networks without retraining.
problem Susceptibility of neural networks to untrustworthy predictions, especially adversarial attacks and out-of-distribution data.
method Geometric Gradient Analysis (GGA) analyzes the geometry of neural network loss landscapes based on saliency maps.
result GGA outperforms existing methods in detecting untrustworthy predictions, including adversarial and out-of-distribution data.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
Deep learning dynamics exhibit anomalous superdiffusion initially, aiding escape from local minima.
problem Understanding the dynamics of learning in deep neural networks.
method Novel analysis of SGD dynamics and loss landscape structure.
result SGD exhibits anomalous superdiffusion initially, transitioning to subdiffusion as learning progresses.
Gradient descent converges to perfect classification in neural nets for non-separable data.
problem Classifying linearly non-separable data using neural networks.
method Analysis of gradient descent dynamics in neural networks with sufficient but not large number of neurons.
result Gradient descent converges to global minima with perfect classification in the landscape of minimization problems.
Proposes using mode connectivity to improve adversarial robustness of neural networks.
problem Improving adversarial robustness of deep neural networks.
method Employing mode connectivity in loss landscapes to study adversarial robustness and propose methods for improvement.
result Path connection learned using limited bonafide data can effectively mitigate adversarial effects while maintaining original accuracy.
We discuss from a geometric point of view the connection between the renormalization group flow for non--linear sigma models and the Ricci flow. This offers new perspectives in providing a geometrical landscape for 2D quantum field theories. In particular we argue that the structure of Ricci flow singularities suggests…
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Study geometric properties of loss functions to understand neural network performance.
problem Understanding the geometric properties of high-dimensional loss functions to improve neural network performance.
method Combine concepts from high-dimensional probability and differential geometry to study curvature properties in lower-dimensional loss representations.
result Mean curvature in the original loss space determines if saddle points appear as minima, maxima, or flat regions.
The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, …
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
problem Understanding geometric principles behind Adam's success in stochastic optimization.
method Introduces GeoAdaLer, an adaptive learning method based on geometric properties.
result Extends interpretability and effectiveness in complex optimization scenarios.
This paper introduces SRPR for robust phase retrieval with smoothed loss functions.
problem Robust phase retrieval from noisy quadratic measurements with corruptions.
method Smoothed robust phase retrieval (SRPR) using convolution-type smoothed loss functions.
result SRPR has no spurious local solutions and benign landscape under corruptions.
In this paper we develop a new framework that captures the common landscape underlying the common non-convex low-rank matrix problems including matrix sensing, matrix completion and robust PCA. In particular, we show for all above problems (including asymmetric cases): 1) all local minima are also globally optimal; 2) …
A longstanding question in superstring/M theory is does it predict supersymmetry below the string scale? We formulate and discuss a necessary condition for this to be true; this is the mathematical conjecture that all stable, compact Ricci flat manifolds have special holonomy in dimensions below eleven. Almost equiva…
Unified geometric framework for quantum states using dual number algebras.
problem Representing quantum states in a geometrically unified way.
method Smooth embeddings into higher-order dual number algebras and algebraic flows.
result Established nilpotent dual algebras as a geometric landscape for quantum kinematics.
Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance…
Persistence landscapes map persistence diagrams into a function space, which may often be taken to be a Banach space or even a Hilbert space. In the latter case, it is a feature map and there is an associated kernel. The main advantage of this summary is that it allows one to apply tools from statistics and machine lea…
In topological data analysis, persistent homology is used to study the "shape of data". Persistent homology computations are completely characterized by a set of intervals called a bar code. It is often said that the long intervals represent the "topological signal" and the short intervals represent "noise". We give ev…
EpiMer merges models by solving Fréchet mean on a Riemannian manifold.
problem Integrating knowledge from multiple models without retraining.
method EpiMer casts model merging as solving the Fréchet mean on a Riemannian manifold, restricting computation to a low-rank subspace.
result EpiMer outperforms flat-geometry methods on image classification tasks.
Data-driven model shows deep learning weights behave like a liquid.
problem Understanding the structure of deep neural network optimization landscapes.
method Statistical mechanics framework to model high-dimensional weight spaces.
result Deep networks' weight spaces are well-connected, not hierarchical, unlike shallow networks.
We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one ne…
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
We study constrained nonconvex optimization problems in machine learning, signal processing, and stochastic control. It is well-known that these problems can be rewritten to a minimax problem in a Lagrangian form. However, due to the lack of convexity, their landscape is not well understood and how to find the stable e…
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
Geometric analysis improves convergence of variational inference.
problem Challenges in analyzing convergence of variational inference due to non-convexity and non-smoothness.
method Exploits exponential family structure and Bregman divergences to geometrically analyze the optimization landscape.
result Establishes non-asymptotic convergence rates for gradient descent algorithms.
Neural collapse occurs in normalized features over a Riemannian manifold.
problem Understanding neural collapse in normalized feature models.
method Simplified multi-class classification task to a nonconvex optimization problem over the Riemannian manifold, analyzing the landscape of critical points.
result The only global minimizers are neural collapse solutions, with all other critical points being strict saddles.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
Mathematical pipeline identifies structural homology of knotted proteins.
problem Quantification and classification of protein structures, especially knotted proteins, require noise-free and complete data.
method Developed a geometric framework using persistent homology to analyze protein structures.
result Persistent homology accurately represents structural homology of knotted proteins and identifies geometric features of protein entanglement.
Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple lay…
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
Unified geometric flows improve deep learning efficiency and simplify neural network topologies.
problem Improving deep learning performance and simplifying neural network structures.
method Proposes a thermodynamically coupled Ricci flow that dynamically adapts parameter space geometry to loss landscape topology, enabling automated singularity resolution and providing entanglement entropy bounds.
result Demonstrates 2.1× convergence acceleration and 63% topological simplification while maintaining O(NlogN) complexity, outperforming Riemannian baselines by 15.2% in few-shot accuracy. New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
A deep learning model organizes RNA graphs to reveal folding patterns and properties.
problem Organizing and understanding the complex folding patterns of RNA secondary structures.
method Geometric scattering autoencoder (GSAE) network for learning graph embeddings.
result GSAE accurately reflects bistable RNA structures and can sample new folding trajectories.
Non-convex optimization with local search heuristics has been widely used in machine learning, achieving many state-of-art results. It becomes increasingly important to understand why they can work for these NP-hard problems on typical data. The landscape of many objective functions in learning has been conjectured to …
This paper organizes recent deep learning theory advances.
problem Lack of theoretical foundations in deep learning.
method Literature review and categorization into six groups.
result Organized recent advances in deep learning theory.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.