Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
New methods using natural gradient for structured optimization.
problem Structured optimization problems.
method Structured second-order methods via natural gradient descent.
result Efficiency demonstrated on non-convex and deep learning problems.
TANGO optimizes models with small learning rates, converging to natural gradient.
problem Optimizing models with small learning rates to converge to natural gradient.
method TANGO, a simple algorithm that converges to natural gradient descent.
result TANGO achieves natural gradient descent with small learning rates.
This work proposes a new method for variational inference using Wasserstein gradient descent.
problem Optimizing variational parameters to match a true posterior distribution.
method Reinterpreting VI as an optimization problem over a variational parameter space, using Wasserstein gradient descent.
result The proposed Wasserstein gradient descent can be seen as a generalization of existing optimization techniques in VI.
Unified framework for gradient descent variants in machine learning.
problem Understanding and comparing various gradient descent methods.
method A unified framework interpreting 6 gradient descent variants.
result Some variants coincide under specific conditions.
Disputes the empirical Fisher approximation for natural gradient descent.
problem The empirical Fisher approximation fails to capture second-order information in general.
method Comparison of empirical Fisher and Fisher information matrices.
result The empirical Fisher does not generally approximate the Fisher or Hessian.
Natural gradient descent speeds up convergence in neural networks, especially with overparameterization.
problem Mitigating the effects of curvature in neural network optimization.
method Analysis of natural gradient descent on nonlinear neural networks with stability conditions.
result Natural gradient descent converges efficiently under specific conditions for overparameterized networks.
NGD improves multivariate Gaussian inference by optimizing Fisher information.
problem Efficiently optimizing multivariate Gaussian models.
method Natural Gradient Descent applied to multivariate Gaussian parameters.
result NGD updates are more efficient for symmetric covariance matrices.
Stochastic NGD approximates Bayesian posterior samples near local minima.
problem Approximating Bayesian uncertainty in model parameters near local minima.
method Develops minibatch natural gradient descent (NGD) and introduces stochastic NGD to preserve Bayesian properties.
result Minibatch NGD's stationary distribution approaches a Bayesian posterior near local minima with small learning rates.
Optimizes graph neural networks using natural gradient descent.
problem Improving efficiency and performance of graph neural networks.
method Employing natural gradient descent to optimize graph neural networks.
result Natural gradient optimization leads to superior performance compared to existing methods.
Paper proposes ZO-NGD for more efficient black-box attacks.
problem Vulnerability of state-of-the-art DNNs to adversarial attacks.
method Zeroth-order natural gradient descent (ZO-NGD) for black-box attacks.
result ZO-NGD achieves significantly lower model query complexities.
Bidirectional whitening improves neural network performance.
problem Improving the efficiency and effectiveness of neural networks.
method Extending whitening process to both forward and backward propagation phases.
result Bidirectional whitening enhances natural gradient descent for better performance.
Unified approach combining gradient descent and multiplicative updates.
problem Combining gradient descent and multiplicative updates for machine learning.
method Introduces hypentropy and a family of matrix-based updates.
result Derives tight regret bounds for the new family of updates.
Natural gradient learning improves synaptic plasticity in spiking neurons.
problem Parametrization dependence leads to inconsistencies in classical synaptic plasticity theories.
method Proposes natural gradient descent in Riemannian geometry for spiking neurons.
result Derives a synaptic learning rule that explains biological phenomena.
Paper proposes a new approach for stochastic gradient descent in probabilistic modeling.
problem Finding optimal predictions in probabilistic models with large step sizes.
method Averaging moment parameters instead of natural parameters for constant-step-size stochastic gradient descent.
result Constant-step-size SGD can lead to better predictions in some cases and always converges in infinite-dimensional models.
Quantum Natural Gradient uses quantum geometry for optimization.
problem Optimizing variational quantum circuits efficiently.
method Quantum generalization of Natural Gradient Descent using Quantum Information Geometry.
result Efficient algorithm for computing metric tensor approximations.
Proposes a new stochastic optimization method for MLR models.
problem Slow convergence of SGD in big data scenarios.
method Dual Stochastic Natural Gradient Descent (DNSGD) based on manifold optimization.
result DNSGD converges and has linear computational complexity.
Physics insights into optimization algorithms using differential equations.
problem Understanding dynamics of optimization algorithms in machine learning.
method Unified framework based on physical systems analysis of popular optimization algorithms.
result Unified analysis applicable to non-convex and non-strongly convex problems.
Gradient descent with random initialization solves phase retrieval problems efficiently.
problem Solving systems of quadratic equations for phase retrieval.
method Gradient descent with random initialization for nonconvex least squares problem.
result Gradient descent achieves near-optimal computational and sample complexities for phase retrieval.
A new optimization method improves deep learning model training speed.
problem Optimizing large models with natural gradient descent.
method Kronecker-factored eigenbasis for diagonal variance approximation.
result Improves optimization speed for deep network architectures.
Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…
Optimal transport natural gradient improves optimization in statistical models.
problem Improving optimization in statistical models with continuous sample spaces.
method Pulling back the Wasserstein metric tensor to a parameter space, creating a Riemannian manifold.
result Natural gradient descent outperforms standard gradient descent in Wasserstein distance optimization.
Extended Kalman Filter is shown to be a gradient descent in trajectory space.
problem Estimating state of dynamical systems from noisy measurements.
method Recovery of extended Kalman filter equations from Amari's natural gradient in trajectory space.
result Extended Kalman Filter is equivalent to natural gradient descent in trajectory space.
We develop a more efficient NGD method for structured parameters.
problem Computational challenges in NGD for structured parameter spaces.
method Local-parameter coordinates to simplify Fisher-matrix computations.
result New structured second-order algorithms and learning methods.
Gradient descent solves robust mean estimation in high dimensions.
problem High-dimensional robust mean estimation in the presence of adversarial outliers.
method Gradient descent with a structural lemma showing near-optimal solutions.
result Gradient descent can solve the robust mean estimation problem directly.
Gradient descent finds linear systems from noisy data.
problem Identifying unknown linear dynamical systems from noisy observations.
method Stochastic gradient descent on maximum likelihood objective.
result Gradient descent efficiently converges to global optimizer in polynomial time and sample complexity.
Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…
Gradient descent struggles with high-dimensional data fitting.
problem Gradient descent struggles with high-dimensional data fitting.
method Gradient descent training of a two-layer neural network on empirical or population risk.
result Gradient descent training may not decrease population risk faster than t−4/(d−2) under mean field scaling. Proposes Coherent Gradients to explain and reduce overfitting in neural networks.
problem Why neural networks generalize well despite fitting random data.
method Hypothesis about gradient dynamics and a modification to gradient descent.
result Supports hypothesis with heuristic arguments and perturbative experiments.
We develop a coordinate-free approach to natural gradient descent for scalable neural networks.
problem First-order optimization methods are sensitive to model parameterization.
method We construct a coordinate-free natural gradient and analyze its invariance properties for K-FAC.
result K-FAC's natural gradient matches the coordinate-free update, maintaining invariance to affine transformations.
Gradient descent can be very slow in escaping saddle points.
problem Gradient descent's slowness in escaping saddle points.
method Gradient descent with natural random initialization and non-pathological functions.
result Gradient descent can take exponential time to escape saddle points.
Improves understanding of stochastic NGVI convergence rates.
problem Lack of knowledge about non-asymptotic convergence rates in stochastic NGVI.
method Proved non-asymptotic convergence rates for conjugate likelihoods and showed implicit optimization for non-conjugate likelihoods.
result First O(T1) non-asymptotic convergence rate for stochastic NGVI in conjugate likelihoods. Convolutional networks can denoise images without training data.
problem Denoising and regularization of images without labeled data.
method Exploiting the structural bias of convolutional generators through gradient descent.
result Early-stopped gradient descent denoises/regularizes images effectively.
Gradient descent achieves exact linear convergence rate for symmetric matrix completion.
problem Low-rank symmetric matrix completion using gradient descent.
method Local analysis of gradient descent for symmetric matrices without additional assumptions.
result Closed-form expression of exact linear convergence rate matches practice.
We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …
Gradient descent training of neural networks leads to solutions close to natural cubic splines.
problem Understanding the implicit bias of gradient descent in neural networks.
method Analysis of gradient descent training for wide neural networks, focusing on the curvature penalty and initialization schemes.
result The solutions of gradient descent training are polyharmonic splines for certain initialization schemes.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
We study alignment in linear neural networks and its relation to gradient descent.
problem Understanding alignment in linear neural networks and its impact on training.
method Defined alignment for fully connected networks, analyzed alignment under gradient descent, and compared gradient descent to projected gradient descent for layer-constrained networks.
result Gradient descent can converge linearly to a global minimum when alignment is invariant, and alignment is impossible with large datasets in layer-constrained networks.
Gradient descent algorithms on manifolds solve control and mean computation problems.
problem Control and mean computation on positive definite Hermitian matrices.
method Riemannian and natural gradient algorithms applied to geodesic distance.
result Efficient algorithms for control and mean computation demonstrated.
The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to get your head around however. The intent of this note is to provide simple intuiti…
Paper shows robustness of gradient descent in matrix sensing despite perturbations.
problem Understanding robustness of gradient descent in matrix sensing.
method Developed perturbed gradient flow to capture noise and improve robustness.
result Gradient descent is robust to perturbations in matrix sensing.
Square-root natural-gradient improves variational inference convergence.
problem Challenges in establishing theoretical convergence guarantees for natural-gradient descent.
method Square-root parameterization for Gaussian covariance.
result Establishes novel convergence guarantees for natural-gradient Gaussian inference.
New method learns good initialization for gradient descent from past solutions.
problem Challenges in solving non-convex optimization problems.
method Learning good initialization rules from previous solutions.
result Our approach performs better than random initialization in various non-convex problems.
A new method for manifold learning uses gradient descent in embedding space with geometric constraints.
problem Learning manifolds in high-dimensional spaces with geometric constraints.
method Discretized gradient flow in the space of embeddings with geometric step length bounds.
result Explicit lower bound for step length in embedding space, applicable to manifold learning.
Optimal control methods achieve significantly smaller regret than previously thought.
problem Optimal control in linear dynamical systems with adversarial changes.
method Online gradient descent and online natural gradient methods.
result Achieves logarithmic regret scaling as O(poly(log T)) instead of O(sqrt(T)).
Unified signSGD and gradient descent analysis for neural networks.
problem Performance of sign-based optimization methods in neural networks.
method Unified analysis of separable smoothness and ℓ∞-smoothness, isolating geometric properties affecting performance. result Sign-based methods are preferable over gradient descent under specific Hessian properties in deep networks.
CBO interprets as SGD, leading to global convergence for nonconvex functions.
problem Understanding and improving gradient-based learning algorithms.
method Interpreting CBO as a stochastic relaxation of SGD.
result CBO provably converges globally to minimizers for nonsmooth nonconvex functions.