The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
This paper addresses the issue of feature importance landscapes in complex images and proposes a regularisation technique to improve network performance.
problem Feature importance landscapes in complex images are not as uniform as assumed, affecting network performance.
method Developed the PILCRO objective to regularize weight configurations, making importance landscapes smoother and more data-driven.
result P-regularised networks have a flat importance landscape, train faster, and perform better in accuracy and robustness.
Theory explains symmetry and saddle points in nonconvex optimization landscapes.
problem Understanding the optimization landscape of nonconvex matrix factorization problems.
method Characterizing stationary points and null spaces via invariant groups.
result Identifies infinitely many nonisolated strict saddle points and global minima.
Entropy-SGD improves deep learning by favoring flat minima in the energy landscape.
problem Training deep neural networks to avoid poorly-generalizing sharp minima.
method Constructs a local-entropy-based objective function to bias SGD towards flat regions.
result Entropy-SGD leads to improved generalization over SGD, as shown by experiments and theoretical underpinnings.
Measures intrinsic dimension of neural network landscapes.
problem Quantifying the difficulty of machine learning problems.
method Train networks in randomly oriented subspaces of varying dimensions.
result Intrinsic dimensions are often smaller than expected.
New method tackles rugged optimization landscapes in contact-rich scenarios.
problem Optimization challenges in dynamic environments with deformable objects.
method Combines Bayesian optimization with semi-local 'leaps' for global search.
result Outperforms gradient-based and gradient-free baselines in simulation and real robot experiments.
SGD with machine learning noise converges to global minimum exponentially fast.
problem Optimizing machine learning models with stochastic gradient descent.
method Analysis of SGD with machine learning noise, focusing on energy landscapes and gradient noise.
result SGD converges to the global minimum exponentially fast under certain conditions.
Paper characterizes optimization landscape of Tucker decomposition.
problem Finding exact Tucker decomposition is a nonconvex optimization problem.
method Characterized the optimization landscape and provided a local search algorithm.
result All local minima are globally optimal if tensor has an exact Tucker decomposition.
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Study optimization landscapes for overcomplete representations, showing benign geometric structures.
problem Optimizing overcomplete representations in high-dimensional data analysis.
method Formulate as ℓ4-norm optimization problems with spherical constraint, analyze geometric properties. result Nonconvex objectives have benign geometric structures, ensuring local search algorithms find target solutions.
Optimizes MMD learning for generative models with theoretical guarantees.
problem Theoretical guarantees for optimizing non-convex MMD objectives.
method Analyzes MMD optimization landscape for specific distributions.
result Gradient-based methods globally minimize MMD objective for certain distributions.
flacco simplifies feature-based landscape analysis for optimization problems.
problem Choosing the best optimizer from a portfolio of algorithms.
method Developed an R-package for feature-based landscape analysis.
result Makes landscape analysis accessible and comprehensible.
Sparse neural networks training is difficult due to optimization failures and energy landscape issues.
problem Training sparse neural networks leads to suboptimal solutions and optimization failures.
method Investigated optimization dynamics and energy landscape in sparse neural networks.
result Sparse neural networks have a linear path with a monotonically decreasing objective from initialization to a good solution, but not from a bad solution.
Deep policy gradient algorithms deviate from their framework's predictions.
problem Understanding the behavior of deep policy gradient algorithms.
method Fine-grained analysis of gradient estimation, value prediction, and optimization landscapes.
result The surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the true gradient.
This work analyzes how entropy affects policy optimization in reinforcement learning.
problem The challenge of designing effective policy optimization algorithms in reinforcement learning.
method New visualizations of the optimization landscape based on randomly perturbing the loss function.
result A policy with higher entropy can make the optimization landscape smoother, enabling larger learning rates.
New function class characterizes loss landscape of deep neural networks without over-parametrization.
problem Complex loss landscape of deep neural networks without over-parametrization.
method Proposed a novel class of functions to characterize loss landscape without over-parametrization.
result Gradient-based optimizers possess theoretical guarantees of convergence under the new function class assumption.
The thesis presents a new perspective on high-dimensional optimization.
problem The failure point of classical optimization methods in high dimensions.
method A distributional view of optimization, focusing on random objective functions and Bayesian Optimization.
result The distributional view explains predictable progress in high-dimensional optimization and provides insights into optimal step size control.
Monotonic Linear Interpolation property in neural networks persists despite non-convexity.
problem Understanding the geometric properties of neural network loss landscapes.
method Tools from differential geometry to analyze the monotonicity of neural network weights.
result Sufficient conditions for the Monotonic Linear Interpolation property under mean squared error.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
New findings on optimization landscape of Toeplitz covariance estimation.
problem Understanding the geometry of the Gaussian maximum-likelihood objective for Toeplitz covariance estimation.
method Overparameterized Carathéodory representation of positive definite Toeplitz covariance matrices, focusing on both amplitudes and frequencies.
result Joint optimization of amplitudes and frequencies leads to a benign population landscape, allowing for global recovery of the true Toeplitz covariance.
New framework shows all local minima are globally optimal in non-convex low-rank problems.
problem Non-convex low-rank problems, including matrix sensing, completion, and robust PCA.
method Developed a new framework to analyze the optimization landscapes of these problems.
result All local minima are also globally optimal and no high-order saddle points exist.
Reward models need more than just accuracy for effective RLHF.
problem The effectiveness of reward models in RLHF is not fully understood.
method An optimization perspective to evaluate reward models.
result Reward models with low reward variance can lead to a flat optimization landscape, hindering performance.
Visualizes movement control optimization landscapes to understand why it's hard and how to make it easier.
problem Understanding and optimizing movement control problems in animation research.
method Novel visualizations of high-dimensional control optimization landscapes.
result Trajectory optimization becomes increasingly ill-conditioned with longer trajectories, while parameterizing control as partial target states can act as an efficient preconditioner.
Designs a non-convex objective function to learn one-hidden-layer neural networks.
problem Learning one-hidden-layer neural networks with Gaussian input and nonnegative label.
method Analytic formula for population risk, landscape design of non-convex objective function G(⋅), stochastic gradient descent. result Stochastic gradient descent on G converges to the global minimum and learns the ground-truth parameters. Visualizes optimization landscapes to understand FCN performance.
problem Understanding why FCNs perform well empirically.
method Visualizing objective functions in 3D space, comparing networks, investigating skip-layer connections, and analyzing loss surfaces.
result Skip-layer connections in FCNs promote flat optimization landscapes, leading to better generalization.
EP approximates Bayesian posterior distributions using gradient descent.
problem Bayesian inference's uncomputable posterior distributions.
method Relates EP to gradient descent on a smoothed energy landscape.
result EP is equivalent to gradient descent on a smoothed energy landscape.
Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.
problem Understanding the impact of over-parameterization on optimization landscapes of shallow neural networks.
method Analyzing a simple ReLU neural network with Gaussian inputs, focusing on optimization properties and landscape changes.
result Over-parameterization makes the objective function one-point strongly convex in most directions, aiding optimization.
Paper explores SNN for learning spectral geometric info from data.
problem Challenges in applying traditional eigensolvers to big data.
method Introduces Spectral Neural Networks (SNN) as an alternative.
result Investigates tradeoffs and optimization landscape of SNN.
EWFM trains continuous flows with only energy evaluations, improving sample quality with fewer computations.
problem Efficiently sampling from complex, high-dimensional Boltzmann distributions using only energy evaluations.
method Energy-Weighted Flow Matching (EWFM) using importance sampling and iterative/annealed training.
result Improved sample quality with up to 3 orders of magnitude fewer energy evaluations compared to existing methods.
We analyze the landscape of empirical risk minimization for high-dimensional models, predicting phase transitions and critical point properties.
problem Understanding the complexity and structure of high-dimensional empirical risk landscapes.
method Using the Kac-Rice formula, we analyze the expected number of critical points and their spectral properties, providing detailed predictions.
result We derive complete topological phase diagrams for the phase retrieval problem, predicting BBP-type transitions and critical point stability.
Gradient ascent solves tensor decomposition efficiently, proving all local maxima are global.
problem Optimizing tensor decomposition problems in machine learning.
method Gradient ascent, Kac-Rice formula, random matrix theory.
result Gradient ascent guarantees solving tensor decomposition problems efficiently.
Persistence landscapes map diagrams into function spaces for statistical and machine learning applications.
problem Mapping persistence diagrams into function spaces for statistical and machine learning.
method Introducing persistence landscapes, weighted persistence landscapes, and Poisson-weighted persistence landscape kernels.
result Persistence landscapes allow for the application of statistical and machine learning tools, and are stable and invertible.
Reviews recent findings on neural network landscapes.
problem Non-convexity of loss functions causing bad landscapes.
method Rigorous geometric analysis and empirical exploration.
result Wide neural nets may have sub-optimal local minima.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
MAML outperforms NAL in diverse task landscapes.
problem Understanding when and how MAML outperforms NAL in various task landscapes.
method Analytical and numerical studies in a linear regression setting with a mixture of easy and hard tasks.
result MAML gains over NAL when there is task hardness discrepancy and optimal solutions of hard tasks are closely packed.
This paper explores machine learning landscapes using molecular energy analogy.
problem Understanding the solution space and nature of predictions in machine learning.
method Analogy with molecular potential energy landscapes to explore machine learning landscapes.
result Emergent properties of machine learning landscapes can be related to molecular structure, thermodynamics, and kinetics.
Integrates uncertainty of loss landscape into stochastic optimization.
problem Improving convergence and generalization in stochastic optimization.
method Incorporates variance of stochastic loss function into momentum updates.
result Improved convergence rates on MNIST and CIFAR-10 datasets.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.
Black holes offer insights into machine learning's loss landscapes.
problem Understanding the loss landscape in machine learning.
method Comparing machine learning loss landscapes to black hole entropy.
result Black holes provide an infinite family of potential landscapes with known minima.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
A novel multi-objective optimization framework improves insurance pricing fairness.
problem Exacerbated trade-offs between competing fairness criteria in insurance pricing using machine learning.
method Proposes a novel multi-objective optimization framework using NSGA-II to jointly optimize accuracy and fairness criteria.
result Consistently achieves a balanced compromise between accuracy and fairness, outperforming single-model approaches.
Efficiently infers graph edges from genetic similarity data in landscape genetics.
problem Inferring unknown graph edges from genetic similarity data in a heterogeneous landscape.
method Developed an efficient first-order optimization method to solve the inverse landscape genetics problem.
result Our method provides fast and reliable convergence, significantly outperforming existing heuristics.
Machine learning speeds up the construction of virus assembly fitness landscapes.
problem Constructing realistic evolutionary fitness landscapes for viruses is computationally expensive.
method Developed a neural network to model virus assembly efficiency from a whole genome/phenotype space.
result Machine learning significantly reduces the computational time for constructing fitness landscapes.
Study energy landscapes in glass models, focusing on Gaussian and spiked-tensor functions.
problem Characterize statistical properties and phase transitions of high-dimensional energy landscapes.
method Developed a Kac-Rice method framework to compute landscape complexity and analyze phase transitions rigorously.
result Characterized the ruggedness and arrangements of local minima in energy landscapes.
LRSAO uses RL to dynamically select and unlearn auxiliary objectives for EA optimization.
problem Optimizing complex landscapes with evolutionary algorithms.
method Local Reinforcement Learning (RL) to dynamically select and unlearn auxiliary objectives.
result Improved optimization complexity from O(n2log(n)/ℓ) to Θ(n2/ℓ2+nlog(n)). Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
Unified model predicts structure of neural network loss landscapes.
problem Understanding the structure of neural network loss landscapes.
method Modeling loss landscape as high-dimensional wedges, analyzing hyperparameters' effects.
result Existence of low-loss subspaces connecting solutions.