LSAM optimizes deep learning training with improved efficiency.
problem Inefficiency in distributed large-batch training with Sharpness-Aware Minimization (SAM).
method Integrates SAM's adversarial steps with an asynchronous distributed sampling strategy.
result Higher final accuracy compared to data-parallel SAM.
FPCA optimizes fairness in target vectors' span.
problem Fairness in principal component analysis for multiple target vectors.
method Non-concave maximization of worst projected target norm using sub-gradient descent.
result Optimization landscape is benign with globally optimal local minima.
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
This paper explores neural network loss landscapes and their effects on generalization.
problem Understanding the structure of neural network loss functions and their impact on generalization.
method Simple filter normalization and various visualization methods to explore loss landscape structure and network architecture effects.
result Visualizations reveal how network architecture and training parameters affect loss landscape curvature and minimizers.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Visualizes optimization landscapes to understand FCN performance.
problem Understanding why FCNs perform well empirically.
method Visualizing objective functions in 3D space, comparing networks, investigating skip-layer connections, and analyzing loss surfaces.
result Skip-layer connections in FCNs promote flat optimization landscapes, leading to better generalization.
Designs a non-convex objective function to learn one-hidden-layer neural networks.
problem Learning one-hidden-layer neural networks with Gaussian input and nonnegative label.
method Analytic formula for population risk, landscape design of non-convex objective function G(⋅), stochastic gradient descent. result Stochastic gradient descent on G converges to the global minimum and learns the ground-truth parameters. This work analyzes how entropy affects policy optimization in reinforcement learning.
problem The challenge of designing effective policy optimization algorithms in reinforcement learning.
method New visualizations of the optimization landscape based on randomly perturbing the loss function.
result A policy with higher entropy can make the optimization landscape smoother, enabling larger learning rates.
The study tests inferences about neural network optimization from linear interpolation of loss landscapes.
problem Understanding the difficulty of neural network optimization problems.
method Linear interpolation of neural network loss landscapes, systematic evaluation of various factors.
result Linear interpolation does not correlate with model performance, challenging prior intuition.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
Expressive quantum circuits are harder to train due to flatter cost landscapes.
problem Designing quantum circuits that are both expressive and trainable.
method Deriving a relationship between expressibility and gradient magnitude, extending barren plateau phenomenon.
result Highly expressive ansätze exhibit flatter cost landscapes, making them harder to train.
Study uses Bayesian Optimization to analyze noise effects in materials research.
problem Optimizing materials with many variables and experimental noise.
method Batch Bayesian Optimization with synthetic data analysis.
result Noise sensitivity varies by problem landscape, impacting optimization outcomes.
Deep learning dynamics exhibit anomalous superdiffusion initially, aiding escape from local minima.
problem Understanding the dynamics of learning in deep neural networks.
method Novel analysis of SGD dynamics and loss landscape structure.
result SGD exhibits anomalous superdiffusion initially, transitioning to subdiffusion as learning progresses.
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Paper shows ERM's suboptimality due to bias, not variance.
problem Understanding why ERM fails to achieve optimal rates.
method Probabilistic and admissibility proofs for ERM in various settings.
result ERM's suboptimality is due to bias, not variance.
New framework reveals thermodynamic principles for LLM training.
problem Understanding the training dynamics of large language models.
method Introducing Neural Thermodynamic Laws (NTL) under river-valley loss landscape assumptions.
result Key thermodynamic quantities and principles naturally emerge in LLM training.
New research shows flat minima in robust loss landscapes correlate with good adversarial robustness.
problem Adversarial training leads to robust overfitting, poor robust generalization.
method Average- and worst-case metrics to measure flatness in robust loss landscapes.
result Flatness in robust loss landscapes correlates with good adversarial robustness.
New algorithm learns convolutional neural networks with overlapping patches.
problem Learning convolutional neural networks with overlapping patches.
method Algorithm draws from isotonic regression and landscape analysis.
result Algorithm works for general class of patches, including common computer vision structures.
Unified framework for neural networks under general input distributions.
problem Training neural networks with non-Gaussian input distributions.
method Designing loss functions with desirable landscape properties for general input distributions.
result Stochastic gradient descent can recover true parameters with global initializations for general input distributions.
Automated model selects best algorithm for continuous problems efficiently.
problem Optimizing continuous black-box problems with limited resources.
method Combining ELA features with machine learning for algorithm selection.
result Average resource requirement is less than half compared to best single solver.
Deep neural networks are optimizable due to their multilayered structure.
problem Understanding why deep neural networks are easily optimizable despite their non-convex loss functions.
method Analysis of a spin glass model of deep neural networks using random matrix theory and algebraic geometry.
result The multilayered structure of deep neural networks leads to fewer stationary points, more clustered minima, and less severe tradeoffs between depth and width of minima.
Machine learning attacks mimic cellular decision-making, revealing new defense mechanisms.
problem Adversarial perturbations fool machine learning models, similar to how ligands prevent correct signaling in cells.
method Formal analogy between neural networks and cellular decision-making models, applying machine learning techniques to study cellular processes.
result Found two regimes in cellular decision-making models, each with a critical point that shapes the loss landscape and defense mechanisms.
Flexible Kernels for Protein Property Prediction
problem Predicting protein properties from sparse experimental data
method Sequence kernels using evolutionary substitution matrices and local linearity
result Data-efficient models of protein property landscapes
Overparametrization improves QNN trainability by reducing spurious local minima.
problem Understanding how overparametrization affects the loss landscape of QNNs.
method Rigorous analysis of overparametrization in QNNs with periodic structure.
result Overparametrization corresponds to a computational phase transition improving QNN trainability.
Persistence landscapes map diagrams into function spaces for statistical and machine learning applications.
problem Mapping persistence diagrams into function spaces for statistical and machine learning.
method Introducing persistence landscapes, weighted persistence landscapes, and Poisson-weighted persistence landscape kernels.
result Persistence landscapes allow for the application of statistical and machine learning tools, and are stable and invertible.
Deep Curvature Suite offers a PyTorch package for neural network curvature analysis.
problem Insufficient use of curvature information in neural networks.
method Implementation of Lanczos algorithm for neural network curvature analysis.
result Our package outperforms existing methods for similar purposes.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
Deep neural networks undergo hierarchical free-energy landscape transitions with increasing data size.
problem Understanding the design space and dynamics of deep neural networks.
method Statistical mechanical approach based on replica method.
result Hierarchical free-energy landscape transitions with ultrametricity, leading to simpler configurations in deeper layers.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Machine learning speeds up the construction of virus assembly fitness landscapes.
problem Constructing realistic evolutionary fitness landscapes for viruses is computationally expensive.
method Developed a neural network to model virus assembly efficiency from a whole genome/phenotype space.
result Machine learning significantly reduces the computational time for constructing fitness landscapes.
Study energy landscapes in glass models, focusing on Gaussian and spiked-tensor functions.
problem Characterize statistical properties and phase transitions of high-dimensional energy landscapes.
method Developed a Kac-Rice method framework to compute landscape complexity and analyze phase transitions rigorously.
result Characterized the ruggedness and arrangements of local minima in energy landscapes.
Unified model predicts structure of neural network loss landscapes.
problem Understanding the structure of neural network loss landscapes.
method Modeling loss landscape as high-dimensional wedges, analyzing hyperparameters' effects.
result Existence of low-loss subspaces connecting solutions.
Researchers improve visualization of neural network loss landscapes.
problem Understanding neural network generalization performance.
method Novel 'jump and retrain' procedure, non-linear dimensionality reduction (PHATE), computational homology.
result Improved visualization and quantification of neural network generalization performance.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Neural networks' energy landscape is surprisingly flat, suggesting minimal structural changes between minima.
problem Understanding the structure of neural network energy landscapes.
method Constructing continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100.
result Paths between minima are essentially flat in both training and test landscapes, implying minimal structural changes.
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular pot…
New sampler tackles complex discrete energy landscapes efficiently.
problem Stagnation in gradient-based discrete samplers for non-convex settings.
method DREXEL sampler with Replica Exchange and Adjusted Metropolis.
result Proves samplers satisfy detailed balance and converge to target distribution.
Exploring loss landscapes of XOR networks reveals complex structures.
problem Understanding the optimization landscape of XOR networks.
method Using molecular science optimization tools, analyzing the number and types of stationary points.
result The landscape of XOR networks becomes more convex with increased regularisation, embedding smaller networks in larger ones.
Discovering quasipotential equations from data using machine learning.
problem Understanding escape mechanisms from metastable states in nonlinear systems.
method Combining neural networks and sparse regression to symbolically reconstruct quasipotential equations.
result Model-unbiased analytical forms of quasipotential discovered directly from data.
Paper presents a more accurate method for nonparametric density estimation using FMMPL and SIR.
problem Improving nonparametric density estimation for complex datasets.
method Finite mixture model of nonparametric density estimation using sampling importance resampling.
result FMMPL provides more accurate results with less space complexity.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
New findings suggest adversarial training does not flatten loss landscapes, challenging current intuition.
problem Understanding and improving generalization in deep learning.
method Loss surface visualization with filter normalization technique.
result Adversarial training does not result in flatter loss landscapes, challenging current intuition.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
PropEn uses matching to create a larger dataset for efficient design optimization.
problem Limited data and complex landscapes in scientific applications.
method PropEn uses a matching approach to implicitly guide design without a discriminator.
result PropEn efficiently approximates the gradient of property improvement within the data distribution.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.