This paper addresses the issue of feature importance landscapes in complex images and proposes a regularisation technique to improve network performance.
problem Feature importance landscapes in complex images are not as uniform as assumed, affecting network performance.
method Developed the PILCRO objective to regularize weight configurations, making importance landscapes smoother and more data-driven.
result P-regularised networks have a flat importance landscape, train faster, and perform better in accuracy and robustness.
MD tree diagnoses model failures using loss landscape metrics.
problem Diagnose model failures without knowing training configuration.
method MD tree based on loss landscape metrics.
result MD tree achieves 87.7% accuracy in dataset transfer tasks, outperforming validation-based approaches.
Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition sta…
This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.
Model explains surprising properties of neural network training landscapes.
problem Understanding the geometry of neural network loss landscapes.
method Developed a simple theoretical model of gradients and Hessians.
result Unified model accounts for 4 surprising properties of neural loss landscapes.
A new loss function α-loss improves classification robustness and calibration.
problem Improving classification robustness and calibration in machine learning.
method Introduces a tunable loss function α-loss, parameterized by α, and analyzes its theoretical and practical properties. result The α-loss function can improve model robustness to label flips and sensitivity to imbalanced classes. Neural collapse occurs in normalized features over a Riemannian manifold.
problem Understanding neural collapse in normalized feature models.
method Simplified multi-class classification task to a nonconvex optimization problem over the Riemannian manifold, analyzing the landscape of critical points.
result The only global minimizers are neural collapse solutions, with all other critical points being strict saddles.
SGD with over-param. makes neural net landscape connected, aiding optimization.
problem Spurious local minima and disconnected landscape in neural networks optimization.
method Stochastic Gradient Descent (SGD) with over-parameterization.
result SGD solutions are connected via a piecewise linear path, making the landscape approximately connected.
Next generation deep neural networks for classification hosted on embedded platforms will rely on fast, efficient, and accurate learning algorithms. Initialization of weights in learning networks has a great impact on the classification accuracy. In this paper we focus on deriving good initial weights by modeling the e…
Sampling strategies significantly affect feature approximations in ELA, impacting classifier accuracy.
problem The impact of sampling strategies on feature approximations in ELA.
method Analysis of feature approximations from different sampling strategies and sample sizes.
result Feature approximations from different sampling strategies do not converge, affecting classifier accuracy.
Gradient descent converges to perfect classification in neural nets for non-separable data.
problem Classifying linearly non-separable data using neural networks.
method Analysis of gradient descent dynamics in neural networks with sufficient but not large number of neurons.
result Gradient descent converges to global minima with perfect classification in the landscape of minimization problems.
Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …
Persistence landscapes map diagrams into function spaces for statistical and machine learning applications.
problem Mapping persistence diagrams into function spaces for statistical and machine learning.
method Introducing persistence landscapes, weighted persistence landscapes, and Poisson-weighted persistence landscape kernels.
result Persistence landscapes allow for the application of statistical and machine learning tools, and are stable and invertible.
PLLay adds topological layers to deep learning models efficiently.
problem Efficiently incorporating topological features into deep learning models.
method Persistence landscapes for differentiable topological features.
result PLLay improves model learnability and robustness.
Analysis of SGD for Gaussian mixture classification using dynamical mean-field theory.
problem Learning dynamics of SGD for a neural network classifying Gaussian mixture.
method Applying dynamical mean-field theory to track SGD dynamics in high dimensions.
result Reveals how SGD navigates the non-convex loss landscape.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
EpiMer merges models by solving Fréchet mean on a Riemannian manifold.
problem Integrating knowledge from multiple models without retraining.
method EpiMer casts model merging as solving the Fréchet mean on a Riemannian manifold, restricting computation to a low-rank subspace.
result EpiMer outperforms flat-geometry methods on image classification tasks.
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
A new loss function α-loss bridges log-loss and 0-1 loss for binary classification.
problem Improving binary classification performance using a tunable loss function.
method Introducing α-loss, proving its margin-based form and classification-calibration, and providing an upper bound on empirical risk. result Empirical and expected risk difference upper bound for logistic regression-based classification.
New method reduces curvature for improved adversarial robustness.
problem Improving the robustness of classifiers against adversarial attacks.
method Proposes a new curvature regularizer to minimize the curvature of the loss surface.
result Adversarial robustness is on par with adversarial training when curvature is minimized.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Machine learning speeds up the construction of virus assembly fitness landscapes.
problem Constructing realistic evolutionary fitness landscapes for viruses is computationally expensive.
method Developed a neural network to model virus assembly efficiency from a whole genome/phenotype space.
result Machine learning significantly reduces the computational time for constructing fitness landscapes.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
Unified model predicts structure of neural network loss landscapes.
problem Understanding the structure of neural network loss landscapes.
method Modeling loss landscape as high-dimensional wedges, analyzing hyperparameters' effects.
result Existence of low-loss subspaces connecting solutions.
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
MAML outperforms NAL in diverse task landscapes.
problem Understanding when and how MAML outperforms NAL in various task landscapes.
method Analytical and numerical studies in a linear regression setting with a mixture of easy and hard tasks.
result MAML gains over NAL when there is task hardness discrepancy and optimal solutions of hard tasks are closely packed.
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection…
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
SimStock learns stock similarities for better investment management.
problem Challenges in identifying similar stocks due to non-stationary financial markets.
method Temporal self-supervised learning framework combining SSL and temporal domain generalization.
result SimStock outperforms existing methods in finding similar stocks.
New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
fSGLD optimizes deep learning by favoring flat regions in the loss landscape.
problem Understanding and improving the behavior and generalization of deep learning algorithms.
method Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD) that biases learning towards flat basins.
result fSGLD targets a flatness-biased Gibbs distribution with explicit excess risk guarantees.
Paper presents a more accurate method for nonparametric density estimation using FMMPL and SIR.
problem Improving nonparametric density estimation for complex datasets.
method Finite mixture model of nonparametric density estimation using sampling importance resampling.
result FMMPL provides more accurate results with less space complexity.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
New findings suggest adversarial training does not flatten loss landscapes, challenging current intuition.
problem Understanding and improving generalization in deep learning.
method Loss surface visualization with filter normalization technique.
result Adversarial training does not result in flatter loss landscapes, challenging current intuition.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Enhanced Particle Swarm Optimization improves ANN classification accuracy and stability.
problem Improving classification accuracy of ANN models.
method Proposes an enhanced Particle Swarm Optimization for ANN training.
result Significant improvement in classification accuracy through stability analysis.
NAS favors wide and shallow cell structures, leading to fast convergence but not necessarily better generalization.
problem Understanding and improving the architectures generated by NAS algorithms.
method Empirical and theoretical study of existing NAS algorithms (DARTS, ENAS) and their architectures.
result Existing NAS algorithms favor wide and shallow cell structures, leading to fast convergence but not necessarily better generalization.
Experimental fractal landscape dynamics observed in emulsions.
problem Understanding anomalous motions in soft glassy materials.
method Quantitative analysis of oil droplet trajectories in dense emulsions.
result Experimental fractal geometry matches computational model of soft glassy dynamics.
Study landscape of non-convex empirical risk with degenerate population risk.
problem Degenerate non-convex population risk in machine learning problems.
method Analyze population risk first, then connect to empirical risk landscape.
result Established correspondence between empirical and population risk critical points.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Study visualizes actor-critic loss landscapes for inventory optimization.
problem Difficulties in solving multi-store dynamic inventory control problems.
method Low-dimensional visualizations of actor loss function.
result Loss landscapes favor optimal policies in reinforcement learning.
We study rough high-dimensional landscapes in which an increasingly stronger preference for a given configuration emerges. Such energy landscapes arise in glass physics and inference. In particular we focus on random Gaussian functions, and on the spiked-tensor model and generalizations. We thoroughly analyze the stati…
Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.
problem Challenges in optimizing models under adversarial training due to loss landscape properties.
method Analytical studies of adversarial loss functions, numerical analyses, PAS strategy.
result Adversarial training impairs optimization, but PAS strategy improves model performance.
LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.