Improved score matching methods for estimating score functions and Hessians without high dimensionality.
problem Estimating score functions and Hessians efficiently in high-dimensional data.
method Implicit score matching and denoising score matching, leveraging Gagliardo-Nirenberg inequalities.
result Achieves convergence rates similar to denoising score matching and estimates Hessians without dimensionality issues.
A new method for training diffusion models using likelihood matching.
problem Training efficient and accurate diffusion models.
method Likelihood Matching approach, quasi-likelihood approximation, score and Hessian estimation.
result Consistent matching of first two transitional moments between diffusion steps.
Unified approach to domain generalization by aligning gradients and Hessians.
problem Developing models that generalize well across unseen domains.
method Moment Alignment, extending transfer measure to DG, aligning derivatives across domains.
result Moment Alignment unifies gradient and Hessian matching approaches, improving generalizability.
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repe…
Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random…
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
Optimizes quadratic bandits with tight Hessian-dependent sample complexity bounds.
problem Understanding optimal sample complexity for quadratic functions.
method Introduces energy allocation and optimal energy spectrum to prove tight lower bounds. Solves for Hessian-independent optimal algorithm.
result Proves optimal Hessian-dependent sample complexities and existence of a universally optimal algorithm.
The paper proves constant rank theorems for special Lagrangian equations.
problem Understanding saddle solutions and Liouville type results for special Lagrangian equations.
method Argument based on saddle solutions and Liouville type results for the special Lagrangian equation.
result Obtained constant rank theorems for saddle solutions to the special Lagrangian equation and the quadratic Hessian equation.
Hessian alignment improves OOD generalization in deep learning.
problem Improving deep learning models' ability to generalize to out-of-distribution data.
method Analyzed Hessian and gradient alignment for domain generalization using recent OOD theory.
result Hessian alignment methods achieve promising performance on various OOD benchmarks.
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
problem Improving generalization in fine-tuned deep neural networks, especially in noisy conditions.
method PAC-Bayesian analysis to identify a Hessian-based distance measure, proving generalization bounds, and developing an algorithm with a generalization error guarantee.
result Hessian-based distance measure correlates well with observed generalization gaps and can match the scale of these gaps in practice.
New averaging technique speeds up Newton method convergence.
problem Superlinear convergence of stochastic Newton methods with noisy Hessians.
method Hessian averaging to reduce noise and maintain superlinear convergence.
result Hessian averaging achieves superlinear convergence with a non-asymptotic rate.
Paper proposes HCDC to improve hyperparameter search efficiency.
problem Poor generalizability of dataset condensation across different hyperparameters.
method HCDC algorithm that matches hyperparameter gradients for synthetic validation dataset.
result HCDC effectively maintains validation-performance rankings of models.
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
Improved online learning for hidden-convex losses achieves optimal regret.
problem Adversarial online learning with nonconvex losses that become convex after reparameterization.
method Algorithmic equivalence between OGD and OMD on convex losses, with Hessian compatibility condition.
result OGD achieves O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) regret for hidden-convex losses, matching optimal rate. New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O ( 1 k 2 ) \mathcal{O}(\frac{1}{k^2}) O ( k 2 1 ) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
Improved graph neural network bounds using graph diffusion matrix.
problem Empirical performance of graph neural networks on real-world graphs.
method Unified model of graph neural networks, focusing on feature diffusion matrix stability.
result Generalization bounds scale with largest singular value of feature diffusion matrix, smaller than prior bounds.
New algorithms tackle complex multi-block optimization problems in machine learning.
problem Non-convex multi-block bilevel optimization with hierarchical sampling challenges.
method Blockwise stochastic variance-reduced methods with parallel speedup.
result Achieves matching complexity to single-block problems with parallel speedup.
The Hessian-vector product has been utilized to find a second-order stationary solution with strong complexity guarantee (e.g., almost linear time complexity in the problem's dimensionality). In this paper, we propose to further reduce the number of Hessian-vector products for faster non-convex optimization. Previous a…
Introduces optimization geometrodynamics for dynamic geometric optimization.
problem Gradient-based optimization methods struggle with changing geometric constraints.
method Optimization geometrodynamics separates invariant and improvable geometric mismatches.
result Dynamic geometric complexity measures the minimum geometric cost to reduce optimization difficulty.
Efficiently approximates higher-order derivatives for generative models.
problem Expensive computation of higher-order derivatives in generative models.
method Rewrite SM objective in terms of directional derivatives and use finite difference for efficient approximation.
result Comparable results to gradient-based methods but significantly more computationally efficient.
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
New metric for probability measures connects physics and geometry.
problem Developing a new metric for probability measures.
method Transport Hessian metric, formulated dynamical systems.
result Connections to physics equations and mathematical models.
Investigates regularity of solutions to complex Hessian equation.
problem Regularity of solutions to complex Hessian equation.
method Analyzes solutions to Dirichlet problem with specific density condition.
result Establishes conditions for regularity of solutions.
Abstract: Survey on quadratic Hessian equations, their properties, and open problems.
problem Understanding quadratic Hessian equations and their solutions.
method Survey and review of existing research.
result Survey of entire solutions, viscosity solutions, and Hessian estimates.
Paper solves Hessian equations on Kähler manifolds.
problem Solving Hessian equations on Kähler manifolds.
method Combines elementary symmetric functions; provides sufficient and necessary condition.
result Generalizes results for Hessian and Hessian quotient equations.
New Hessian estimates for heat equations on manifolds.
problem Estimating Hessian matrices for heat-type equations on Riemannian manifolds.
method Using Bismut-Stroock Hessian formula, with explicit coefficients and delay/growth rate functions.
result Novel backward weak Harnack inequality and precise pointwise Hessian estimates for eigenfunctions.
This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only O ~ ( ε − 3.5 ) \mathcal{\tilde{O}}(ε^{-3.5}) O ~ ( ε − 3.5 ) stochastic gradien…
Study calculates Hessian of Busemann function on Damek-Ricci spaces.
problem Calculating Hessian of Busemann function on Damek-Ricci spaces.
method Calculates eigenvalues of Hessian and proves positive definiteness.
result Hessian of Busemann function is positive definite.
The paper shows infinitely many components in Floer Hessians space.
problem Understanding the structure of Floer Hessians.
method Proving the existence of infinitely many connected components.
result Proves infinitely many connected components in Floer Hessians space.
The paper describes flat Hessian metrics on surfaces and their potentials.
problem Understanding Hessian metrics on surfaces.
method Theoretical description and explicit construction using integrable systems.
result Explicit construction of potentials for flat Hessian metrics on surfaces.
The paper quantizes Hessian structures on R^2 using KV-algebras.
problem Quantizing Hessian structures on a 2D space.
method Deformation quantization within Koszul-Vinberg algebras.
result Established links between deformation theory and Hessian geometry.
We prove that, in dimensions greater than 2, the generic metric is not a Hessian metric and find a curvature condition on Hessian metrics in dimensions greater than 3. In particular we prove that the forms used to define the Pontryagin classes in terms of the curvature vanish on a Hessian manifold. By contrast all anal…
Superintegrable systems on curved manifolds found to have Hessian structures.
problem Characterizing superintegrable systems on curved manifolds.
method Identifying and computing Hessian coordinates for superintegrable systems.
result Examples of superintegrable systems in 2D and 3D have natural Hessian coordinates.
Curved Frobenius manifolds link to Hessian metrics in geometry.
problem Understanding curved Frobenius manifolds and their relation to Hessian metrics.
method Analyzing the relationship between curved Frobenius structures and Hessian metrics on spaces with non-vanishing curvature.
result Consistent curved Frobenius structures on constant curvature spaces are linked to Hessian metrics.
New rigidity results for a generalized Ricci-Hessian equation on manifolds.
problem Understanding rigidity in generalized Ricci-Hessian equations on manifolds.
method Proving new rigidity results related to a generalized Ricci-Hessian equation.
result New rigidity results for the generalized Ricci-Hessian equation on Riemannian manifolds.
Study classifies 3D Hessian manifolds, proving their topology.
problem Global topology of 3D Hessian manifolds.
method Proved structure and analyzed Betti numbers.
result Complete topological classification of 3D Hessian manifolds.
Gaussians as noise in NCE lead to exponentially bad conditioning, hindering its efficiency.
problem Exponential conditioning of Hessian in NCE with Gaussian noise.
method Using Gaussian as the noise distribution in NCE.
result Gaussian noise in NCE leads to exponentially bad conditioning of the loss Hessian.
Criterion for solvability of complex 2-Hessian equation on compact Kähler manifolds.
problem Solvability of complex 2-Hessian equation on compact Kähler manifolds.
method Nakai--Moishezon-type criterion associated with the complex 2-Hessian equation.
result Criterion equivalent to existence of a smooth 2-admissible representative in complex dimension three.
Establishes a concavity property for positive Hessian quotient operators.
problem Analyzing positive Hessian quotient operators on Riemannian manifolds.
method Proves a special concavity property and a Jacobi inequality.
result Proves a Jacobi inequality for symmetric tensors.
Estimates for complex Hessian equations on Hermitian manifolds.
problem Establishing estimates for solutions to complex Hessian equations.
method Using concavity inequality for complex sum-of-Hessian operators.
result Second-order estimates for admissible solutions on Hermitian manifolds.
Derives Hessian estimates for Lagrangian mean curvature equation.
problem Lagrangian mean curvature equation with supercritical phase and bounded second derivatives.
method Derives a priori interior Hessian estimates.
result Hessian estimates for Lagrangian mean curvature equation.
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
A new unbiased Hessian estimator for expectation-based objectives.
problem Estimating Hessian for objectives with non-reparameterizable nodes.
method GO Hessian estimator for expectation-based objectives.
result GO Hessian provides unbiased and low-variance estimation of Hessian.
Constructs homogeneous Kähler structures on tangent bundles of Hessian manifolds.
problem Creating Kähler structures on tangent bundles of Hessian manifolds.
method Endowing Hessian manifolds with Kähler structures using group actions and homothetic vector fields.
result Homogeneous conformally Kähler structures on tangent bundles of selfsimilar Hessian manifolds.
Extends dimension reduction to data-driven settings without gradients.
problem Gradient-based dimension reduction limitations in data-driven settings.
method Score ratio matching framework, tailored parameterization, regularization, eigenvalue deflation.
result Outperforms standard score-matching for problems with low-dimensional structure.
Paper proves inequalities on Hermitian manifolds with applications to bounded solutions.
problem Establishing mixed Hessian inequalities on Hermitian manifolds.
method Weak convergence theorem of complex Hessian operators and general mixed Hessian inequality.
result Existence of bounded solutions of complex Hessian equations.
A selfsimiar manifold is a Riemannian manifold ( M , g ) \left(M,g\right) ( M , g ) endowed with a homothetic vector field ξ ξ ξ . We characterize global selfsimilar manifolds and describe the structure of local selfsimilar manifolds. We prove that any selfsimilar manifold with a potential homothetic vector field is a conical Riemannian ma…
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.