AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
New research shows flat minima in robust loss landscapes correlate with good adversarial robustness.
problem Adversarial training leads to robust overfitting, poor robust generalization.
method Average- and worst-case metrics to measure flatness in robust loss landscapes.
result Flatness in robust loss landscapes correlates with good adversarial robustness.
Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, …
New model shows SGD can prefer sharp or flat solutions based on label noise.
problem Understanding SGD's preference for flat or sharp solutions during training.
method Solved an analytically solvable model to explore SGD behavior.
result Data distribution determines sharpness at convergence; isotropic label noise leads to flat minimum preference.
We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our re…
DPlis improves privacy in deep learning models by smoothing loss functions.
problem Privacy leakage in deep learning models trained on private data and low model performance.
method DPlis constructs a smooth loss function to favor noise-resilient models.
result DPlis effectively boosts model quality and training stability under privacy constraints.
Looped transformers outperform standard transformers in complex reasoning tasks due to a specific loss landscape geometry.
problem Understanding why looped transformers outperform standard transformers in complex reasoning tasks.
method Explained through loss landscape geometry, distinguishing between U-shaped and V-shaped valleys, and proposing SHIFT training strategy.
result Looped transformers' recursive architecture induces a River-V-Valley landscape, leading to better loss convergence and complex pattern learning.
This paper addresses the issue of feature importance landscapes in complex images and proposes a regularisation technique to improve network performance.
problem Feature importance landscapes in complex images are not as uniform as assumed, affecting network performance.
method Developed the PILCRO objective to regularize weight configurations, making importance landscapes smoother and more data-driven.
result P-regularised networks have a flat importance landscape, train faster, and perform better in accuracy and robustness.
Many image processing tasks involve image-to-image mapping, which can be addressed well by fully convolutional networks (FCN) without any heavy preprocessing. Although empirically designing and training FCNs can achieve satisfactory results, reasons for the improvement in performance are slightly ambiguous. Our study i…
A new approach to quantum machine learning circuits reduces training difficulties.
problem Challenges in training deep quantum circuits due to flat training landscapes.
method Variable structure approach (VAns) to build ansatzes, applying rules for gate growth and removal.
result VAns successfully mitigates trainability and noise-related issues, improving performance in various applications.
DEO uses gradient information to escape saddle points in neural networks.
problem Training deep neural networks struggles with flat regions and saddle points.
method Dimer-Enhanced Optimization (DEO) uses gradient information to estimate curvature and escape saddle points.
result DEO improves training efficiency and performance compared to standard first-order methods.
Paper finds exact Hessian sharpness in deep matrix factorization.
problem Understanding the geometry of loss landscapes in deep matrix factorization.
method Presented the first exact expression for Hessian maximum eigenvalue.
result Spectral-norm balance is a sufficient condition for flatness in deep matrix factorization.
Researchers improve NCE by addressing its flat loss landscape issues.
problem NCE's poor performance due to an ill-behaved loss landscape.
method Introduced eNCE with an exponential loss and normalized gradient descent.
result Proven that landscape issues arise from inappropriate noise distribution.
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenval…
We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. O…
fSGLD optimizes deep learning by favoring flat regions in the loss landscape.
problem Understanding and improving the behavior and generalization of deep learning algorithms.
method Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD) that biases learning towards flat basins.
result fSGLD targets a flatness-biased Gibbs distribution with explicit excess risk guarantees.
FP-BMA improves generalization by encouraging flat posteriors in Bayesian Model Averaging.
problem Lack of flat posterior in approximate Bayesian inference methods hinders effective Bayesian Model Averaging.
method Proposes Flat Posterior-aware Bayesian Model Averaging (FP-BMA) and Flat Posterior-aware Bayesian Transfer Learning schemes.
result FP-BMA successfully captures flat posteriors, improving generalization performance.
A longstanding question in superstring/M theory is does it predict supersymmetry below the string scale? We formulate and discuss a necessary condition for this to be true; this is the mathematical conjecture that all stable, compact Ricci flat manifolds have special holonomy in dimensions below eleven. Almost equiva…
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
New method accelerates neural network training by focusing on flat directions.
problem Improving neural network training speed and stability.
method Bulk-SGD, interpolated gradient methods.
result Updates along the Dominant subspace can accelerate convergence but compromise stability.
Variational quantum computing faces a flat optimization landscape problem.
problem Barren Plateaus (BP) in optimization landscapes.
method Theoretical and heuristic methods to understand and mitigate BPs.
result All algorithm components can lead to BPs if not well-suited.
Proposes a method to sample from flat basins of posterior distributions in Bayesian deep learning.
problem Sampling from multi-modal posterior distributions leads to overfitting due to trapping in bad modes.
method Introduces an auxiliary guiding variable to bias MCMC sampling towards flat basins of the energy landscape.
result The method converges faster and outperforms existing methods in sampling from flat basins of the posterior.
Reward models need more than just accuracy for effective RLHF.
problem The effectiveness of reward models in RLHF is not fully understood.
method An optimization perspective to evaluate reward models.
result Reward models with low reward variance can lead to a flat optimization landscape, hindering performance.
Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.
problem Challenges in optimizing models under adversarial training due to loss landscape properties.
method Analytical studies of adversarial loss functions, numerical analyses, PAS strategy.
result Adversarial training impairs optimization, but PAS strategy improves model performance.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
SAM minimizes loss sharpness, improving adversarial transferability.
problem Improving adversarial transferability of deep neural networks.
method Evaluating surrogate models trained with seven minimizers, focusing on loss sharpness and flat neighborhoods.
result SAM minimizes loss sharpness, leading to better adversarial transferability.
New research challenges the flatness-generalization link in deep neural networks.
problem The correlation between flatness of the loss landscape and generalization in deep neural networks is questioned.
method The study examines various flatness measures and popular SGD variants, finding some break the flatness-generalization link. It proposes using logP(f), a global quantity, as a predictor of generalization. result The log of Bayesian prior upon initialization, logP(f), is a significantly more robust predictor of generalization than flatness measures. Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
problem Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
method Identifying entropic barriers arising from the interplay between curvature variations along low-loss paths and noise in optimization dynamics.
result Curvature-induced entropic forces bias noisy dynamics back toward the endpoints, explaining the confinement and connectivity of solutions.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
DASH improves ensemble generalizability by encouraging diverse, flat loss landscapes.
problem Improving generalization and robustness of deep ensembles.
method DASH promotes diversity and flatness in deep ensembles by encouraging base learners to move towards low-loss regions of minimal sharpness.
result DASH improves ensemble generalizability, as demonstrated by extensive empirical evidence.
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce mi…
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
MD tree diagnoses model failures using loss landscape metrics.
problem Diagnose model failures without knowing training configuration.
method MD tree based on loss landscape metrics.
result MD tree achieves 87.7% accuracy in dataset transfer tasks, outperforming validation-based approaches.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
Study geometric properties of loss functions to understand neural network performance.
problem Understanding the geometric properties of high-dimensional loss functions to improve neural network performance.
method Combine concepts from high-dimensional probability and differential geometry to study curvature properties in lower-dimensional loss representations.
result Mean curvature in the original loss space determines if saddle points appear as minima, maxima, or flat regions.
New algorithms improve neural network generalization by finding flat minima.
problem Finding better generalization in neural networks through flat minima.
method Developed Entropy-SGD and Replicated-SGD algorithms to maximize flatness in the loss function.
result Consistently improved generalization error for various deep learning architectures.
Large learning rates cause parameter instability, leading to better generalization.
problem Understanding why deep neural networks perform well despite operating outside the traditional stability regime.
method Analyzing the effect of large learning rates on the orientation of Hessian eigenvectors and parameter exploration.
result Large learning rates induce parameter instability, leading to better generalization through exploration of flatter regions of the loss landscape.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mi…
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
The pursuit of explaining and improving generalization in deep learning has elicited efforts both in regularization techniques as well as visualization techniques of the loss surface geometry. The latter is related to the intuition prevalent in the community that flatter local optima leads to lower generalization error…
Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition sta…