arXiv research

Analyzes the structure and rank of neural network Hessians.

This paper uncovers the low-rank structure of neural network Hessians.

New perspective on CNNs using Hessian maps reveals their structure.

In this article we show that every geodesic is rank one and the Hessian of Busemann functions is positive definite for a harmonic Damek-Ricci space, a two step solvable Lie group with a left invariant metric. Moreover, the eigenspace of the Hessian of Busemann functions on a Hadamard manifold $(M,g)$ corresponding to e…

Locally conformally Hessian manifolds are dense in radiant ones of rank 1.

Analyzes Hessian spectrum for neural networks near optimal learning.

Study classifies special Hessian rank 2 hypersurfaces in 4D space.

The paper proves constant rank theorems for special Lagrangian equations.

The study classifies Hessian rank 1 hypersurfaces in dimensions 2, 3, and 4.

New geometric structures derived from Hessian metrics and tensors.

No non-product Hessian rank 1 affine homogeneous hypersurfaces exist in dimensions 5 and above.

The study proves the existence of $k$-convex hypersurfaces for specific curvature equations.

New method proves asymptotic normality for matrix sensing problems.

In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repe…

Study on affine surfaces with specific algebraic properties.

Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to …

The goal of tensor completion is to fill in missing entries of a partially known tensor (possibly including some noise) under a low-rank constraint. This may be formulated as a least-squares problem. The set of tensors of a given multilinear rank is known to admit a Riemannian manifold structure, thus methods of Rieman…

The completion of low rank matrices from few entries is a task with many practical applications. We consider here two aspects of this problem: detectability, i.e. the ability to estimate the rank $r$ reliably from the fewest possible random entries, and performance in achieving small reconstruction error. We propose a …

A new quasi-Newton method uses cubic regularization to avoid saddle points in deep learning.

New ACV method speeds up CV in high dimensions with approximate low-rank data.

Flat minima lead to better generalization in low-rank matrix recovery models.

SGD's training dynamics align with Hessian and gradient spectra in high-dimensional classification tasks.

New formulas for Riemannian gradient and Hessian on manifold metrics.

In a complete Riemannian manifold $(M, g)$ if the hessian of a real valued function satisfies some suitable conditions then it restricts the geometry of $(M, g)$. In this paper we characterize all compact rank-1 symmetric spaces, as those Riemannian manifolds $(M, g)$ admitting a real valued function $u$ such that the …

New method learns shared structures in non-linear tasks.

Equivalent formulations for low-rank matrix optimization are proven.

Study uncovers complex critical points in tensor decomposition.

The paper explores the geometric structure of cost functions in multiple dimensions.

We study stability and local minimizing properties of $L^p$- norms of Riemannian curvature tensor denoted by $\mathcal{R}_p$ by variational methods. We compute the Hessian of $\mathcal{R}_p$ at compact rank 1 symmetric spaces and prove that they are stable for $\mathcal{R}_p$ for certain values of p > 2. A similar resu…

New findings challenge the use of flatness measures in neural networks.

Paper proposes HCDC to improve hyperparameter search efficiency.

Random Matrix Theory explains loss surface Hessians in neural networks.

Paper proposes a new flatness measure for neural networks to improve generalization.

Reinforcement Learning (RL) algorithms allow artificial agents to improve their action selections so as to increase rewarding experiences in their environments. Deep Reinforcement Learning algorithms require solving a nonconvex and nonlinear unconstrained optimization problem. Methods for solving the optimization probl…

Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …

Muon optimizer outperforms GD in neural networks.

Let $Ω\subset\mathbb R^n$ be a Lipschitz domain. Given $1\leq p<k\leq n$ and any $u\in W^{2,p}(Ω)$ belonging to the little Hölder class $c^{1,α}$, we construct a sequence $u_j$ in the same space with $\operatorname{rank}D^2u_j<k$ almost everywhere such that $u_j\to u$ in $C^{1,α}$ and weakly in $W^{2,p}$. This result i…

New algorithms estimate Jacobian matrices for large-scale machine learning.

A new algorithm reduces the time and space complexity for multinomial logistic bandits.

FedLoRU improves FL efficiency by using low-rank updates.

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

EpiMer merges models by solving Fréchet mean on a Riemannian manifold.

SGD updates align with a low-rank subspace but do not lead to further loss reduction.

Novel Newton method for large-scale kernel methods using random features.

New insights into matrix factorization show strict saddles have bounded eigenvalues.

The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.

New metric for probability measures connects physics and geometry.

Investigates regularity of solutions to complex Hessian equation.