HessFormer enables distributed Hessian computation for large models.
problem Limited Hessian computation for large deep learning models.
method Distributed stochastic Lanczos quadrature algorithm.
result Investigated Hessian spectral density of a 70 billion parameter model.
A selfsimiar manifold is a Riemannian manifold (M,g) endowed with a homothetic vector field ξ. We characterize global selfsimilar manifolds and describe the structure of local selfsimilar manifolds. We prove that any selfsimilar manifold with a potential homothetic vector field is a conical Riemannian ma…
ASTRA improves TDA by more accurately approximating iHVP.
problem Improving insights into training data attribution.
method ASTRA uses EKFAC-preconditioner on Neumann series iterations to accurately approximate iHVP.
result Improving iHVP approximation significantly improves TDA performance.
New SGD algorithm finds critical points faster with second-order corrections.
problem Finding critical points in non-convex optimization efficiently.
method Uses Hessian-vector products to correct momentum bias in SGD.
result Finds ε-critical points in O(ε−3) time. A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Stochastic Variance-Reduced Cubic regularization (SVRC) algorithms have received increasing attention due to its improved gradient/Hessian complexities (i.e., number of queries to stochastic gradient/Hessian oracles) to find local minima for nonconvex finite-sum optimization. However, it is unclear whether existing SVR…
A new algorithm reduces communication rounds for distributed convex optimization.
problem Efficiently solving convex optimization problems in distributed systems.
method Proposes a stochastic Newton algorithm for homogeneous distributed stochastic convex optimization.
result Reduces the number and frequency of communication rounds compared to existing methods.
The Hessian-vector product has been utilized to find a second-order stationary solution with strong complexity guarantee (e.g., almost linear time complexity in the problem's dimensionality). In this paper, we propose to further reduce the number of Hessian-vector products for faster non-convex optimization. Previous a…
New method for efficient sketching of gradients and Hessians.
problem Memory constraints in training machine learning models.
method A novel framework for scalable gradient and HVP sketching tailored for modern hardware.
result Theoretical guarantees and practical applications in training data attribution and Hessian spectrum analysis.
WoodFisher improves neural network compression efficiency and accuracy.
problem Efficiently estimating inverse Hessian for neural network optimization.
method WoodFisher: a method to compute a faithful and efficient estimate of the inverse Hessian.
result WoodFisher significantly outperforms state-of-the-art methods for pruning neural networks.
Paper studies Hessian quotient equations in warped product manifolds.
problem Analyzing Hessian quotient equations in warped product manifolds.
method Using standard degree theory and a priori estimates.
result Existence of star-shaped compact hypersurface solutions.
A new unbiased Hessian estimator for expectation-based objectives.
problem Estimating Hessian for objectives with non-reparameterizable nodes.
method GO Hessian estimator for expectation-based objectives.
result GO Hessian provides unbiased and low-variance estimation of Hessian.
The paper studies efficient Hessian fitting methods for stochastic optimization.
problem Efficient Hessian fitting for stochastic optimization.
method Preconditioned Stochastic Gradient Descent (PSGD) method and Lie groups.
result Hessian fitting problem is strongly convex in certain Lie groups.
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…
HTE improves PINNs for high-dimensional, high-order PDEs by reducing computational cost and memory usage.
problem Challenges in solving high-dimensional, high-order PDEs with PINNs due to computational cost and memory constraints.
method Introduces Hutchinson Trace Estimation (HTE) to transform Hessian matrix calculations into Hessian vector products (HVP), reducing computational cost and memory usage.
result HTE significantly reduces memory consumption and computational cost, enabling faster and more efficient solution of high-dimensional and high-order PDEs.
In this paper we study a Ricci-Hessian type manifold (M,g,φ,f,λ) which is closely related to the construction of almost Ricci soliton realized as a warped product. We classify certain classes of the Ricci-Hessian type manifolds and derive some implications for almost Ricci solitons and generalized m--qu…
Kunneth formula derived for flat affine manifolds and applied to Hessian metrics.
problem Cohomology of flat affine manifolds and Hessian metrics.
method Proved Künneth formula for Dolbeault--Koszul cohomology of flat affine manifolds and applied to Hessian metrics.
result Kunneth formula for Hessian metrics on products and manifolds with hyperbolic manifolds.
The paper studies convexity of products of squared Euclidean distances.
problem Convexity of products of squared Euclidean distances.
method Proved a convexity principle and applied it to products of squared distances, computed Hessian-positive regions and exact convexity levels.
result Computed exact convexity and quasiconvexity truncation levels for the two-centre model.
We analyze the Hessian spectra of large models up to 100B parameters.
problem Accurate Hessian spectra of large foundation models are difficult to obtain.
method We use shard-local finite-difference Hessian vector products and stochastic Lanczos quadrature.
result We produce the first large-scale spectral density estimates of foundation models.
Derives formulas from Green function Hessian assumption.
problem Deriving formulas from Green function Hessian assumption.
method Assumption on Hessian of Green function leads to monotonicity formulas.
result Explicit examples of manifolds satisfying assumption.
New method computes affine normal directions efficiently for sparse polynomials.
problem Computing affine normal directions is computationally expensive in high dimensions.
method Reduces third-order tensor contraction to matrix-free formulation using log-determinant gradient.
result Scalable implementations with near-linear scaling in dimension and sparsity.
We show that if a compact Kaehler manifold M of non-negative Ricci curvature admits closed Fedosov star product then the reduced Lie algebra of holomorphic vector fields on M is reductive. This comes in pair with the obstruction previously found by La Fuente-Gravy. More generally we consider the squared norm of Cah…
This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.
problem Efficiently solving large-scale bilevel optimization problems with gradient-based methods.
method Constructing low-dimensional approximate Krylov subspaces with the Lanczos process to approximate the Hessian inverse vector product.
result Demonstrates a O(ε−1) convergence rate and efficiency in synthetic and deep learning tasks. Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random…
The target space geometry of abelian vector multiplets in N=2 theories in four and five space-time dimensions is called special geometry. It can be elegantly formulated in terms of Hessian geometry. In this review, we introduce Hessian geometry, focussing on aspects that are relevant for the special geometrie…
The paper introduces new functionals and equations for complex vector bundles.
problem Complex vector bundle equations and their solutions.
method Introducing generalized Donaldson's functionals and discussing their Euler-Lagrange equations.
result Uniqueness of solutions to complex vector bundle equations.
BPS solutions of 5-dimensional supergravity correspond to certain gradient flows on the product M x N of a quaternionic-Kaehler manifold M of negative scalar curvature and a very special real manifold N of dimension n >=0. Such gradient flows are generated by the `energy function' f = P^2, where P is a (bundle-valued) …
Constructs homogeneous Kähler structures on tangent bundles of Hessian manifolds.
problem Creating Kähler structures on tangent bundles of Hessian manifolds.
method Endowing Hessian manifolds with Kähler structures using group actions and homothetic vector fields.
result Homogeneous conformally Kähler structures on tangent bundles of selfsimilar Hessian manifolds.
The paper quantizes Hessian structures on R^2 using KV-algebras.
problem Quantizing Hessian structures on a 2D space.
method Deformation quantization within Koszul-Vinberg algebras.
result Established links between deformation theory and Hessian geometry.
We propose a reduction for non-convex optimization that can (1) turn an stationary-point finding algorithm into an local-minimum finding one, and (2) replace the Hessian-vector product computations with only gradient computations. It works both in the stochastic and the deterministic settings, without hurting the algor…
Study classifies 3D Hessian manifolds, proving their topology.
problem Global topology of 3D Hessian manifolds.
method Proved structure and analyzed Betti numbers.
result Complete topological classification of 3D Hessian manifolds.
The study finds starshaped compact hypersurfaces in warped products with curvature estimates.
problem Finding starshaped compact hypersurfaces in warped product manifolds.
method Deriving global curvature estimates and interior second order a priori estimates for solutions to associated equations.
result Existence of starshaped compact hypersurfaces in warped product manifolds.
No non-product Hessian rank 1 affine homogeneous hypersurfaces exist in dimensions 5 and above.
problem Identifying non-product Hessian rank 1 affine homogeneous hypersurfaces in higher dimensions.
method Developed a normal form for hypersurfaces under the affine group, up to order ≤ n+5, in any dimension n ≥ 2.
result Non-existence of non-product Hessian rank 1 affine homogeneous hypersurfaces in dimensions 5 and above.
New formulas for Riemannian gradient and Hessian on manifold metrics.
problem Evaluate Riemannian gradient and Hessian for various metrics on manifolds.
method Explicit formulas derived from Levi-Civita connection and projection.
result Derives new metrics and optimization frameworks on manifolds.
Improves understanding of neural network predictions using influence functions.
problem Challenges in understanding neural network predictions.
method Utilized NTK theory to calculate influence functions for over-parameterized neural networks.
result Proved that the approximation error of IF can be arbitrarily small in the over-parameterized regime.
Study geometric properties of loss functions to understand neural network performance.
problem Understanding the geometric properties of high-dimensional loss functions to improve neural network performance.
method Combine concepts from high-dimensional probability and differential geometry to study curvature properties in lower-dimensional loss representations.
result Mean curvature in the original loss space determines if saddle points appear as minima, maxima, or flat regions.
Shampoo optimizes preconditioners for faster convergence in machine learning.
problem Improving convergence speed in machine learning optimization.
method Explicit connection between Shampoo's Kronecker product approximation and optimal matrix approximations.
result The square of Shampoo's approximation is equivalent to a single power iteration step for optimal Kronecker product approximation.
Curved Frobenius manifolds link to Hessian metrics in geometry.
problem Understanding curved Frobenius manifolds and their relation to Hessian metrics.
method Analyzing the relationship between curved Frobenius structures and Hessian metrics on spaces with non-vanishing curvature.
result Consistent curved Frobenius structures on constant curvature spaces are linked to Hessian metrics.
Proves triviality and nonexistence of gradient Ricci solitons as warped metrics.
problem Proving triviality and nonexistence of gradient Ricci solitons as warped metrics.
method Proved through the construction of gradient Ricci solitons as warped products and studying Ricci-Hessian type manifolds.
result Gradient Ricci solitons are trivial and non-existent as warped metrics.
Establishes statistical and computational bounds for influence diagnostics.
problem Identifying influential datapoints or subsets in machine learning models.
method Finite-sample statistical bounds and computational complexity for influence functions and approximate maximum influence perturbations.
result Established statistical and computational guarantees for influence diagnostics.
This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only O~(ε−3.5) stochastic gradien…
This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…
New algorithms tackle complex multi-block optimization problems in machine learning.
problem Non-convex multi-block bilevel optimization with hierarchical sampling challenges.
method Blockwise stochastic variance-reduced methods with parallel speedup.
result Achieves matching complexity to single-block problems with parallel speedup.
Paper develops a splitting principle for RCD spaces, extending manifold properties.
problem Understanding splitting properties in RCD spaces.
method General analytic splitting principle for RCD spaces.
result Spaces with suitable functions have splitting properties.
Develops efficient quasi-Newton methods for training deep neural networks.
problem Training deep neural networks with large-scale Hessian matrices.
method Approximates Hessian as block-diagonal Kronecker product of smaller matrices, applies damping.
result Outperforms or matches state-of-the-art methods in autoencoder models.
In this paper we study the space of solutions to an overdetermined linear system involving the Hessian of functions. We show that if the solution space has dimension greater than one, then the underlying manifold has a very rigid warped product structure. We obtain a uniqueness result for prescribing the Ricci curvatur…