New method shows Hessian estimator from random samples converges to true Hessian on complex manifolds.
problem Uncertainty in Hessian estimator accuracy on complex manifolds with boundaries and nonuniform sampling.
method Locally fitting quadratic polynomials, rigorous theoretical analysis under mild conditions.
result The Hessian estimator asymptotically converges to the true Hessian, even near boundaries.
The paper estimates curvature for a specific type of equations.
problem Estimating curvature for Hessian type equations.
method Establishing curvature estimates for a class of Hessian type equations.
result Curvature estimates for Hessian type equations have been successfully established.
We analyze the Hessian spectra of large models up to 100B parameters.
problem Accurate Hessian spectra of large foundation models are difficult to obtain.
method We use shard-local finite-difference Hessian vector products and stochastic Lanczos quadrature.
result We produce the first large-scale spectral density estimates of foundation models.
Characterizes Hessian eigenspectra for realistic nonlinear models.
problem Understanding Hessian eigenspectra in realistic nonlinear models.
method Deterministic equivalent techniques from random matrix theory.
result Hessian can have qualitatively different spectral behaviors.
The Hessian Topology is a subject having interesting relations with several areas, for instance, differential geometry, implicit differential equations, analysis and singularity theory. In this article we study the problem of realization of a real plane curve as the Hessian curve of a smooth function. The plane curves …
New perspective on CNNs using Hessian maps reveals their structure.
problem Understanding the nature of Convolutional Neural Networks (CNNs).
method Developed a framework using Toeplitz representation of CNNs to reveal Hessian structure and prove rank bounds.
result Proved that the Hessian rank of CNNs grows as the square root of the number of parameters.
New method for efficient sketching of gradients and Hessians.
problem Memory constraints in training machine learning models.
method A novel framework for scalable gradient and HVP sketching tailored for modern hardware.
result Theoretical guarantees and practical applications in training data attribution and Hessian spectrum analysis.
This research analyzes and accelerates score-based diffusion models using discretization and Hessian information.
problem Theoretical foundations and convergence analysis of score-based diffusion models.
method Investigation of various discretization schemes, including Euler, exponential integrators, and midpoint randomization. Proposal of an accelerated sampler based on local linearization method.
result Hessian-based approach achieves faster convergence rates of order $\widetilde{\mathcal{O}}\left(\frac{1}{\varepsilon}
ight)$, significantly improving upon vanilla diffusion models.
HA-SME models SGD dynamics with Hessian info for better escaping behaviors.
problem Capturing the escaping behaviors of SGD from stationary points.
method HA-SME, a novel SDE with Hessian info in drift and diffusion.
result HA-SME achieves best approximation error and recovers SGD dynamics for quadratics.
New insights into Hessian structure of neural networks reveal two forces.
problem Understanding the Hessian structure of neural networks.
method Analyzing the static and dynamic forces, comparing limit distributions using random matrix theory.
result The Hessian structure arises from a combination of static and dynamic forces, with C being a primary driver. New algorithms estimate Hessians using random directions for faster stochastic optimization.
problem Efficiently estimating Hessians for stochastic optimization.
method Generalized Hessian estimators using random directions and noisy function measurements.
result Asymptotically unbiased estimators with lower bias for more measurements.
New Bethe-Hessian method improves community detection in sparse networks.
problem Detect communities in sparse networks efficiently.
method Spectral clustering using the Bethe-Hessian matrix.
result Bethe-Hessian consistently estimates block number above Kesten-Stigum threshold.
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
Hessian alignment improves OOD generalization in deep learning.
problem Improving deep learning models' ability to generalize to out-of-distribution data.
method Analyzed Hessian and gradient alignment for domain generalization using recent OOD theory.
result Hessian alignment methods achieve promising performance on various OOD benchmarks.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O(ε−1.5) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O(ε−1.5) iterations for ε-accurate stationary point. A new spline method for manifold learning using Hessian-based curvature penalties.
problem Learning manifolds with curvature penalties in high dimensions.
method Generalizes thin-plate splines to flat manifolds using Hessian matrices, minimizing square error with curvature constraints.
result Existence and uniqueness of the spline solution, expressed as Green's functions and Hessian approximations.
The paper addresses geometric analysis on non-compact Riemannian manifolds, proving Calderón-Zygmund inequalities.
problem Proving Calderón-Zygmund inequalities on non-compact Riemannian manifolds without positive injectivity radius.
method Probabilistic tools, Hessian formulas, and Bismut type representations for heat semigroups.
result The paper proves the Calderón-Zygmund inequality for 1<p<2 under a lower Ricci curvature bound, and for p>2 under additional curvature conditions. NeCPD improves online tensor decomposition using SGD with Hessian analysis and NAG.
problem Efficiently decompose multi-way tensors in online data processing.
method NeCPD solver based on SGD with Hessian analysis and NAG.
result NeCPD provides more accurate results than existing methods.
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…
This work connects the Hessian to the decision boundary complexity in neural networks.
problem Understanding the decision boundary complexity in high-dimensional input space.
method Characterizing the decision boundary using the Hessian top eigenvectors and analyzing the number of outliers.
result The number of outliers in the Hessian spectrum is proportional to the complexity of the decision boundary.
Recently several methods were proposed for sparse optimization which make careful use of second-order information [10, 28, 16, 3] to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coo…
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
problem Improving generalization in fine-tuned deep neural networks, especially in noisy conditions.
method PAC-Bayesian analysis to identify a Hessian-based distance measure, proving generalization bounds, and developing an algorithm with a generalization error guarantee.
result Hessian-based distance measure correlates well with observed generalization gaps and can match the scale of these gaps in practice.
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, w…
Established in the 30's, Schauder {\it a priori} estimates are among the most classical and powerful tools in the analysis of problems ruled by 2nd order elliptic PDEs. Since then, a central problem in regularity theory has been to understand Schauder type estimates fashioning particular borderline scenarios. In such c…
We establish variational formulas for Ricci upper and lower bounds, as well as a derivative formula for the Ricci curvature. As applications, constant curvature manifolds, Einstein manifolds and Ricci parallel manifolds are identified, respectively, with different integral-differential formulas and semigroup inequaliti…
The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how to coordinate the accuracy in the gradient and Hessian to yield a superlinear ra…
Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.
problem Optimizing non-convex functions like deep learning models using second-order methods.
method Nyström-approximated curvature for stochastic optimization of large-scale empirical risk minimization.
result The proposed method achieves performance competitive with state-of-the-art first-order and stochastic quasi-Newton methods.
We propose a sample efficient stochastic variance-reduced cubic regularization (Lite-SVRC) algorithm for finding the local minimum efficiently in nonconvex optimization. The proposed algorithm achieves a lower sample complexity of Hessian matrix computation than existing cubic regularization based methods. At the heart…
Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…
New methods optimize functions faster with less gradient accuracy needed.
problem Optimizing complex functions with limited gradient accuracy.
method Hessian averaging and adaptive gradient sampling methods.
result Improved convergence rates for various function types.
The understanding of the dynamics of the velocity gradients in turbulent flows is critical to understanding various non-linear turbulent processes. The pressure-Hessian and the viscous-Laplacian govern the evolution of the velocity-gradients and are known to be non-local in nature. Over the years, several simplified dy…
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
Transformers use a unique Hessian structure that differs from classical networks, affecting optimization.
problem Understanding the unique optimization landscape of Transformers.
method Theoretical Hessian analysis of a single self-attention layer in Transformers.
result Transformers have a highly non-linear Hessian structure, distinguishing them from classical networks.
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data that is widely used in observational sciences. In its classic form, ICA relies on modeling the data as linear mixtures of non-Gaussian independent sources. The maximization of the corresponding likelihood is a challen…
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
The paper solves PDEs from matrices with orthogonal columns, linking them to Hessian metrics and symmetric spaces.
problem Solving third order PDEs for strictly convex smooth functions.
method Geometric methods using Hessian metrics and symmetric spaces.
result Explicit solutions and a family of non-generic solutions with applications in Poisson geometry and Kahler structures.
Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the computations and/or to implicitly implement a form of statistical regularization. In this …
New metric for probability measures connects physics and geometry.
problem Developing a new metric for probability measures.
method Transport Hessian metric, formulated dynamical systems.
result Connections to physics equations and mathematical models.
Investigates regularity of solutions to complex Hessian equation.
problem Regularity of solutions to complex Hessian equation.
method Analyzes solutions to Dirichlet problem with specific density condition.
result Establishes conditions for regularity of solutions.
Abstract: Survey on quadratic Hessian equations, their properties, and open problems.
problem Understanding quadratic Hessian equations and their solutions.
method Survey and review of existing research.
result Survey of entire solutions, viscosity solutions, and Hessian estimates.
Wide neural networks become linear, with constant tangent kernel, due to Hessian scaling.
problem Understanding the linearity of large non-linear models and the tangent kernel.
method Analyzing the scaling properties of the Hessian matrix of neural networks as their width increases.
result The constancy of the tangent kernel is due to the scaling properties of the Hessian matrix.
Paper solves Hessian equations on Kähler manifolds.
problem Solving Hessian equations on Kähler manifolds.
method Combines elementary symmetric functions; provides sufficient and necessary condition.
result Generalizes results for Hessian and Hessian quotient equations.
While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and…
New Hessian estimates for heat equations on manifolds.
problem Estimating Hessian matrices for heat-type equations on Riemannian manifolds.
method Using Bismut-Stroock Hessian formula, with explicit coefficients and delay/growth rate functions.
result Novel backward weak Harnack inequality and precise pointwise Hessian estimates for eigenfunctions.
Improved sampling guarantees for underdamped Langevin Monte Carlo without restrictive assumptions.
problem Sampling from unnormalized densities with improved guarantees and acceleration.
method Novel analysis relaxing assumptions on log-Sobolev inequality and Hessian smoothness, using Rényi discretization bounds.
result First KL divergence guarantees for ULMC without Hessian smoothness under strong log-concavity.
Study calculates Hessian of Busemann function on Damek-Ricci spaces.
problem Calculating Hessian of Busemann function on Damek-Ricci spaces.
method Calculates eigenvalues of Hessian and proves positive definiteness.
result Hessian of Busemann function is positive definite.
The paper shows infinitely many components in Floer Hessians space.
problem Understanding the structure of Floer Hessians.
method Proving the existence of infinitely many connected components.
result Proves infinitely many connected components in Floer Hessians space.
The paper describes flat Hessian metrics on surfaces and their potentials.
problem Understanding Hessian metrics on surfaces.
method Theoretical description and explicit construction using integrable systems.
result Explicit construction of potentials for flat Hessian metrics on surfaces.