arXiv research

Better Hessian approximations improve influence function attributions in deep learning.

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ε$-approximate seco…

Newton's method converges linearly for stable Hessians, even with approximations.

Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.

New trust-region algorithms improve ML performance with indefinite Hessians.

We analyze the Hessian spectra of large models up to 100B parameters.

New structure in neural network Hessians explains outliers, improving subspace approximation.

Apollo improves nonconvex stochastic optimization efficiency.

Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…

In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repe…

Efficiently learns RBMs using covariance estimates and adaptive learning rates.

This paper proves subsampled Newton methods work for high-dimensional data.

The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how to coordinate the accuracy in the gradient and Hessian to yield a superlinear ra…

New Hessian-free method improves bilevel optimization for meta-learning.

The most fruitful approach to studying low energy soliton dynamics in field theories of Bogomol'nyi type is the geodesic approximation of Manton. In the case of vortices and monopoles, Stuart has obtained rigorous estimates of the errors in this approximation, and hence proved that it is valid in the low speed regime. …

ASTRA improves TDA by more accurately approximating iHVP.

Paper proposes a new Hessian-aware zeroth-order optimization for improving black-box adversarial attacks.

Improved neural network ensembles using Stein Variational Newton updates.

Study deepnet Hessians, revealing spiked spectrum with SGD and sample size effects.

We consdier in dimension four weakly convergent sequences of approximate biharmonic maos into sphere with bi-tension fields bounded in $L^p$ for some $p>1$. We prove an energy identity that accounts for the loss of Hessian energies by the sum of Hessian energies over finitely many nontrivial biharmonic maps on $\mathbb…

HA-SME models SGD dynamics with Hessian info for better escaping behaviors.

Enhances influence functions for deep models without costly Hessian inversion.

Shampoo optimizes preconditioners for faster convergence in machine learning.

Improved robustness in optimization methods using second-order information.

We consider in dimension four weakly convergent sequences of approximate biharmonic maps to a Riemannian manifold with bi-tension fields bounded in $L^p$ for $p>\frac43$. We prove an energy identity that accounts for the loss of hessian energies by the sum of hessian energies over finitely many nontrivial biharmonic ma…

NHGD solves bilevel optimization problems with reduced computational time.

Proposes a Quasi-Newton trust region method for policy optimization in reinforcement learning.

A parallel optimization method for convex functions using Hessian sketching and debiasing.

Let $(X,ω)$ be a compact Kähler manifold of dimension $n$ and fix $m\in \mathbb{N}$ such that $1\leq m \leq n$. We prove that any $(ω,m)$-sh function can be approximated from above by smooth $(ω,m)$-sh functions. A potential theory for the complex Hessian equation is also developed which generalizes the classical pluri…

SNS accelerates Sinkhorn algorithm with sparse Newton iterations.

D2SRM solves complex PDEs using deep learning.

Recently several methods were proposed for sparse optimization which make careful use of second-order information [10, 28, 16, 3] to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coo…

New algorithms estimate Hessians using random directions for faster stochastic optimization.

Modular method simplifies curvature computation in neural nets.

New methods optimize functions faster with less gradient accuracy needed.

Develops efficient quasi-Newton methods for training deep neural networks.

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…

New methods use Kronecker-factored approximations for faster deep learning optimization.

Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to …

Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with ${O}(ε^{-1.5})$ complexity.

A new spline method for manifold learning using Hessian-based curvature penalties.

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…

Enhances SMC² with Hessian info for more efficient posterior approximation.

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …

New distributed algorithm for second-order optimization with reduced Hessian computation.

Improved SVRC algorithm reduces complexity for nonconvex optimization.

Improved continual learning for neural networks with BN layers using K-FAC extension.

Analyzes the structure and rank of neural network Hessians.