Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
Characterizes Hessian eigenspectra for realistic nonlinear models.
problem Understanding Hessian eigenspectra in realistic nonlinear models.
method Deterministic equivalent techniques from random matrix theory.
result Hessian can have qualitatively different spectral behaviors.
A new mutual information optimization method using self-supervised binary contrastive learning.
problem Improving self-supervised contrastive learning for better model performance.
method Proposes a novel loss function for contrastive learning that optimizes mutual information in positive and negative pairs.
result The proposed method outperforms state-of-the-art self-supervised contrastive frameworks on various benchmark datasets.
Lower bounds on eigenspectrum show rich action spaces force polynomial regret in linear bandits.
problem Understanding the minimum eigenvalue growth in linear bandits with rich action sets.
method Non-asymptotic lower bound on eigenspectrum of design matrix.
result Minimum eigenvalue of expected design matrix grows as Ω ( n ) Ω(\sqrt{n}) Ω ( n ) for sub-linear regret. Self-attention networks localize when eigenspectrum variance is small.
problem Self-attention mechanisms can lead to rank and entropy collapses, reducing model expressivity and trainability.
method Characterized attention localization using query-key eigenspectrum variance.
result Small eigenspectrum variance prevents both rank and entropy collapses, improving model performance.
We present an original and novel method based on random matrix approach that enables to distinguish the respective role of temporal autocorrelations inside given time series and cross correlations between various time series. The proposed algorithm is based on properties of Wigner eigenspectrum of random matrices inste…
Data structure affects deep learning performance, study finds.
problem Understanding why deep learning performs poorly on typical datasets.
method Analyzed input correlation matrices and network Hessians, developed PAC-Bayes bounds.
result Sloppy eigenspectra in input data correlate with poor deep learning performance.
SAT improves adversarial training by smoothing the loss landscape through curriculum learning.
problem Adversarial training sacrifices clean accuracy for robustness and suffers from large generalization error.
method SAT uses curriculum learning to smooth the adversarial loss landscape, improving both clean and robust accuracy.
result SAT models improve clean and robust accuracy significantly compared to adversarial training and other baselines.
ImageNet pre-training has been regarded as essential for training accurate object detectors for a long time. Recently, it has been shown that object detectors trained from randomly initialized weights can be on par with those fine-tuned from ImageNet pre-trained models. However, the effects of pre-training and the diff…
This article investigates the eigenspectrum of the inner product-type kernel matrix p K = { f ( x i T x j / p ) } i , j = 1 n \sqrt{p} \mathbf{K}=\{f( \mathbf{x}_i^{\sf T} \mathbf{x}_j/\sqrt{p})\}_{i,j=1}^n p K = { f ( x i T x j / p ) } i , j = 1 n under a binary mixture model in the high dimensional regime where the number of data n n n and their dimension p p p are both large and comparable. Based on…
New method preserves spectral clustering performance under aggressive sparsification and quantization.
problem Maintaining spectral clustering performance with sparse and quantized data.
method Random matrix theory applied to eigenspectrum changes under sparsification and quantization.
result Spectral clustering performance is preserved even with aggressive sparsification and quantization.
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
New metric for probability measures connects physics and geometry.
problem Developing a new metric for probability measures.
method Transport Hessian metric, formulated dynamical systems.
result Connections to physics equations and mathematical models.
Investigates regularity of solutions to complex Hessian equation.
problem Regularity of solutions to complex Hessian equation.
method Analyzes solutions to Dirichlet problem with specific density condition.
result Establishes conditions for regularity of solutions.
Abstract: Survey on quadratic Hessian equations, their properties, and open problems.
problem Understanding quadratic Hessian equations and their solutions.
method Survey and review of existing research.
result Survey of entire solutions, viscosity solutions, and Hessian estimates.
Paper solves Hessian equations on Kähler manifolds.
problem Solving Hessian equations on Kähler manifolds.
method Combines elementary symmetric functions; provides sufficient and necessary condition.
result Generalizes results for Hessian and Hessian quotient equations.
New Hessian estimates for heat equations on manifolds.
problem Estimating Hessian matrices for heat-type equations on Riemannian manifolds.
method Using Bismut-Stroock Hessian formula, with explicit coefficients and delay/growth rate functions.
result Novel backward weak Harnack inequality and precise pointwise Hessian estimates for eigenfunctions.
SGD in linear regression overfits but performs well due to bias-variance trade-off.
problem Understanding overfitting in SGD for linear regression.
method Constant-stepsize SGD with iterate averaging or tail averaging, analyzing full eigenspectrum of data covariance matrix.
result Sharp excess risk bounds revealing bias-variance decomposition for SGD in linear regression.
Study calculates Hessian of Busemann function on Damek-Ricci spaces.
problem Calculating Hessian of Busemann function on Damek-Ricci spaces.
method Calculates eigenvalues of Hessian and proves positive definiteness.
result Hessian of Busemann function is positive definite.
The paper shows infinitely many components in Floer Hessians space.
problem Understanding the structure of Floer Hessians.
method Proving the existence of infinitely many connected components.
result Proves infinitely many connected components in Floer Hessians space.
The paper describes flat Hessian metrics on surfaces and their potentials.
problem Understanding Hessian metrics on surfaces.
method Theoretical description and explicit construction using integrable systems.
result Explicit construction of potentials for flat Hessian metrics on surfaces.
The paper quantizes Hessian structures on R^2 using KV-algebras.
problem Quantizing Hessian structures on a 2D space.
method Deformation quantization within Koszul-Vinberg algebras.
result Established links between deformation theory and Hessian geometry.
We prove that, in dimensions greater than 2, the generic metric is not a Hessian metric and find a curvature condition on Hessian metrics in dimensions greater than 3. In particular we prove that the forms used to define the Pontryagin classes in terms of the curvature vanish on a Hessian manifold. By contrast all anal…
Superintegrable systems on curved manifolds found to have Hessian structures.
problem Characterizing superintegrable systems on curved manifolds.
method Identifying and computing Hessian coordinates for superintegrable systems.
result Examples of superintegrable systems in 2D and 3D have natural Hessian coordinates.
Curved Frobenius manifolds link to Hessian metrics in geometry.
problem Understanding curved Frobenius manifolds and their relation to Hessian metrics.
method Analyzing the relationship between curved Frobenius structures and Hessian metrics on spaces with non-vanishing curvature.
result Consistent curved Frobenius structures on constant curvature spaces are linked to Hessian metrics.
New rigidity results for a generalized Ricci-Hessian equation on manifolds.
problem Understanding rigidity in generalized Ricci-Hessian equations on manifolds.
method Proving new rigidity results related to a generalized Ricci-Hessian equation.
result New rigidity results for the generalized Ricci-Hessian equation on Riemannian manifolds.
Study classifies 3D Hessian manifolds, proving their topology.
problem Global topology of 3D Hessian manifolds.
method Proved structure and analyzed Betti numbers.
result Complete topological classification of 3D Hessian manifolds.
Criterion for solvability of complex 2-Hessian equation on compact Kähler manifolds.
problem Solvability of complex 2-Hessian equation on compact Kähler manifolds.
method Nakai--Moishezon-type criterion associated with the complex 2-Hessian equation.
result Criterion equivalent to existence of a smooth 2-admissible representative in complex dimension three.
Establishes a concavity property for positive Hessian quotient operators.
problem Analyzing positive Hessian quotient operators on Riemannian manifolds.
method Proves a special concavity property and a Jacobi inequality.
result Proves a Jacobi inequality for symmetric tensors.
Estimates for complex Hessian equations on Hermitian manifolds.
problem Establishing estimates for solutions to complex Hessian equations.
method Using concavity inequality for complex sum-of-Hessian operators.
result Second-order estimates for admissible solutions on Hermitian manifolds.
Derives Hessian estimates for Lagrangian mean curvature equation.
problem Lagrangian mean curvature equation with supercritical phase and bounded second derivatives.
method Derives a priori interior Hessian estimates.
result Hessian estimates for Lagrangian mean curvature equation.
A new unbiased Hessian estimator for expectation-based objectives.
problem Estimating Hessian for objectives with non-reparameterizable nodes.
method GO Hessian estimator for expectation-based objectives.
result GO Hessian provides unbiased and low-variance estimation of Hessian.
Constructs homogeneous Kähler structures on tangent bundles of Hessian manifolds.
problem Creating Kähler structures on tangent bundles of Hessian manifolds.
method Endowing Hessian manifolds with Kähler structures using group actions and homothetic vector fields.
result Homogeneous conformally Kähler structures on tangent bundles of selfsimilar Hessian manifolds.
Paper proves inequalities on Hermitian manifolds with applications to bounded solutions.
problem Establishing mixed Hessian inequalities on Hermitian manifolds.
method Weak convergence theorem of complex Hessian operators and general mixed Hessian inequality.
result Existence of bounded solutions of complex Hessian equations.
A selfsimiar manifold is a Riemannian manifold ( M , g ) \left(M,g\right) ( M , g ) endowed with a homothetic vector field ξ ξ ξ . We characterize global selfsimilar manifolds and describe the structure of local selfsimilar manifolds. We prove that any selfsimilar manifold with a potential homothetic vector field is a conical Riemannian ma…
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
The paper studies efficient Hessian fitting methods for stochastic optimization.
problem Efficient Hessian fitting for stochastic optimization.
method Preconditioned Stochastic Gradient Descent (PSGD) method and Lie groups.
result Hessian fitting problem is strongly convex in certain Lie groups.
Study solves complex Hessian equation on Hermitian manifolds.
problem Solving Hessian equations on Hermitian manifolds with mixed structure.
method Derive a priori estimates and solve Dirichlet problem under conditions.
result Solvability of the Dirichlet problem for mixed Hessian equations.
Study confirms Chern's conjecture on compact Hessian manifolds and classifies their topologies.
problem Global topological constraints and structural properties of compact Hessian manifolds.
method Novel fibration and splitting theorems, Chern's conjecture, Hitchin systems, Cheng-Yau solution.
result Topological classification of complete Hessian surfaces and closed orientable Hessian 3-manifolds.
Derives concavity inequality and estimates for k k k -Hessian equations.
problem Interior estimates and curvature estimates for k k k -Hessian equations. method Concavity inequality and semi-convexity condition.
result Interior estimates and Liouville-type result for semi-convex solutions.
Paper discusses solving generalized Hessian inequalities with various operators.
problem Finding global solutions to generalized Hessian inequalities.
method Analyzes various Hessian operators and provides conditions for global solvability.
result Provides necessary and sufficient conditions for global solvability of generalized Hessian inequalities.
Bounds on Hessian of heat equation coupled with Ricci flow.
problem Estimating the Hessian of a solution to the conjugate heat equation coupled with Ricci flow.
method Obtained upper bounds for the Hessian.
result Local and global upper bounds for the Hessian of a positive solution.
Hessian alignment improves OOD generalization in deep learning.
problem Improving deep learning models' ability to generalize to out-of-distribution data.
method Analyzed Hessian and gradient alignment for domain generalization using recent OOD theory.
result Hessian alignment methods achieve promising performance on various OOD benchmarks.
New perspective on CNNs using Hessian maps reveals their structure.
problem Understanding the nature of Convolutional Neural Networks (CNNs).
method Developed a framework using Toeplitz representation of CNNs to reveal Hessian structure and prove rank bounds.
result Proved that the Hessian rank of CNNs grows as the square root of the number of parameters.
Paper studies Hessian quotient equations in warped product manifolds.
problem Analyzing Hessian quotient equations in warped product manifolds.
method Using standard degree theory and a priori estimates.
result Existence of star-shaped compact hypersurface solutions.
Study the Hessian geometry of an ideal gas in a centrifuge.
problem Understanding the Hessian geometry of an ideal gas in a centrifuge.
method Investigate the Hessian geometry associated with an ideal gas in a spherical centrifuge, using the action of the Euclidean rotation group.
result The Hessian geometry of a spherical rigid body is isometric to a hyperbolic space in the high angular velocity limit.
New Hessian estimators for Riemannian manifolds with reduced bias.
problem Estimating Hessians on Riemannian manifolds with reduced bias and computational efficiency.
method Introducing new stochastic zeroth-order Hessian estimators using O ( 1 ) O(1) O ( 1 ) function evaluations. result Achieved a bias bound of order O ( γ δ 2 ) O(γδ^2) O ( γ δ 2 ) for analytic real-valued functions. New algorithm adds Hessian regularization to improve neural network robustness.
problem Improving neural network robustness against adversarial attacks.
method Proposes an efficient algorithm to train neural networks with Hessian operator-norm regularization.
result Hessian operator-norm regularization increases neural network robustness over input gradient regularization.