arXiv research

New Hessian estimates for heat equations on manifolds.

New Bethe-Hessian method improves community detection in sparse networks.

Analyzes Hessian spectrum for neural networks near optimal learning.

New insights into Hessian structure of neural networks reveal two forces.

On a constraint manifold we give an explicit formula for the Hessian matrix of a cost function that involves the Hessian matrix of a prolonged function and the Hessian matrices of the constraint functions. We give an explicit formula for the case of the orthogonal group ${\bf O}(n)$ by using only Euclidean coordinates …

Enhances influence functions for deep models without costly Hessian inversion.

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …

Characterizes Hessian eigenspectra for realistic nonlinear models.

Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the lim…

Random Matrix Theory explains loss surface Hessians in neural networks.

Paper finds exact Hessian sharpness in deep matrix factorization.

A family of probability distributions parametrized by an open domain $Λ$ in $R^n$ defines the Fisher information matrix on this domain which is positive semi-definite. In information geometry the standard assumption has been that the Fisher information matrix tensor is positive definite defining in this way a Riemannia…

A new hybrid Newton algorithm improves convergence in logistic regression.

New insights into learning rates and batch sizes for neural networks using random matrix theory.

SNS accelerates Sinkhorn algorithm with sparse Newton iterations.

SGD's training dynamics align with Hessian and gradient spectra in high-dimensional classification tasks.

New method proves asymptotic normality for matrix sensing problems.

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…

We propose a sample efficient stochastic variance-reduced cubic regularization (Lite-SVRC) algorithm for finding the local minimum efficiently in nonconvex optimization. The proposed algorithm achieves a lower sample complexity of Hessian matrix computation than existing cubic regularization based methods. At the heart…

The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.

This paper uncovers the low-rank structure of neural network Hessians.

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ε$-approximate seco…

Flat minima lead to better generalization in low-rank matrix recovery models.

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…

Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.

A parallel optimization method for convex functions using Hessian sketching and debiasing.

Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to …

Training avoids edge of stability by aligning Jacobian matrices.

A new optimisation method efficiently scales Hessian-vector products for neural networks.

New Hessian-free method improves bilevel optimization for meta-learning.

The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalu…

The completion of low rank matrices from few entries is a task with many practical applications. We consider here two aspects of this problem: detectability, i.e. the ability to estimate the rank $r$ reliably from the fewest possible random entries, and performance in achieving small reconstruction error. We propose a …

We revisit the landscape of the simple matrix factorization problem. For low-rank matrix factorization, prior work has shown that there exist infinitely many critical points all of which are either global minima or strict saddles. At a strict saddle the minimum eigenvalue of the Hessian is negative. Of interest is whet…

A method to reduce Hessian matrix calculation cost in gradient-based meta-learning.

Apollo improves nonconvex stochastic optimization efficiency.

A new spline method for manifold learning using Hessian-based curvature penalties.

Hessian alignment improves OOD generalization in deep learning.

Better Hessian approximations improve influence function attributions in deep learning.

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one o…

Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.

New formulas for Riemannian gradient and Hessian on manifold metrics.

Proposes a new method to initialize neural networks by estimating global curvature of weights.

New framework for higher-order singular-value derivatives of rectangular matrices.

Muon optimizer outperforms GD in neural networks.

EiGLasso speeds up sparse Kronecker-sum covariance estimation.

The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues. We observe that for wider networks, minimizing the loss with the gradient descent optimization maneuvers through surfaces o…

In distributed optimization and distributed numerical linear algebra, we often encounter an inversion bias: if we want to compute a quantity that depends on the inverse of a sum of distributed matrices, then the sum of the inverses does not equal the inverse of the sum. An example of this occurs in distributed Newton's…

New methods use Kronecker-factored approximations for faster deep learning optimization.