Efficiently learns RBMs using covariance estimates and adaptive learning rates.
problem Learning RBMs using standard methods is computationally expensive.
method Uses Hessian approximations and MCMC samples for covariance estimation, resulting in adaptive learning rates.
result Improves efficiency of learning RBMs compared to standard methods.
New method improves online covariance estimation for SGD.
problem Improving online covariance estimation for SGD.
method Proposes a de-biased covariance estimator that eliminates second-order derivatives.
result Achieves a convergence rate of n ( α − 1 ) / 2 log n n^{(α-1)/2} \sqrt{\log n} n ( α − 1 ) /2 log n , outperforming existing methods. New structure in neural network Hessians explains outliers, improving subspace approximation.
problem Explaining outliers in the spectrum of deepnet Hessians.
method Identified a two-level structure in the Hessian, showing it's not a covariance but a second moment matrix.
result Shows the means of gradients have an additive two-way structure, leading to outliers in the spectrum.
Calculates metrics and geodesics for symplectic forms.
problem Computing metrics and geodesics for symplectic forms.
method Computed the Levi-Civita connection, described geodesics, and computed the covariant Hessian.
result Formula for the covariant Hessian of an energy functional.
Two new covariance estimators for ROOT-SGD improve statistical inference.
problem Uncertainty measurement for ROOT-SGD's normal distribution estimator.
method Developed two covariance estimators: plug-in and Hessian-free.
result Hessian-free estimator is asymptotically consistent and Hessian-free.
Derives formulas from Green function Hessian assumption.
problem Deriving formulas from Green function Hessian assumption.
method Assumption on Hessian of Green function leads to monotonicity formulas.
result Explicit examples of manifolds satisfying assumption.
EiGLasso speeds up sparse Kronecker-sum covariance estimation.
problem Sparse Kronecker-sum inverse covariance estimation challenges in scalability and parameter identification.
method Newton's method combined with eigendecomposition of sample and feature graphs, approximating Hessian for speed.
result Two to three orders-of-magnitude speed-up on simulated and real-world data.
This paper investigates factors influencing SGD minima.
problem Understanding the factors that influence the minima found by SGD.
method Examined learning rate, batch size, Hessian, and gradient covariance; used stochastic differential equations to model SGD.
result The ratio of batch size to learning rate is a main factor in SGD dynamics.
In this brief survey, we will remark the interaction among the Hessian tensor on a semi-Riemannian manifold and some of the several questions in Lorentzian (and also in semi-Riemannian) geometry where this 2-covariant tensor is involved. In particular, we deal with the characterization of Killing vector fields and the …
The paper optimizes regret using covariance between costs and decisions.
problem Optimizing expected regret in decision-making problems.
method Developed derivative theory of covariance regret functional, derived Gâteaux derivative, and extended to constrained optimization.
result Gradient of covariance regret is the cost covariance matrix, with implications for portfolio optimization.
Develops calculus for tamed Dirichlet spaces using measure theory.
problem Defines calculus for measure spaces with Dirichlet forms.
method Introduces first and second order calculus on tamed Dirichlet spaces.
result Defines various geometric objects on tamed Dirichlet spaces.
Paper controls shape stability in infinite Riemannian manifolds.
problem Characterizing optimal shapes in infinite-dimensional Riemannian manifolds.
method Uses Riemannian manifold framework and mean curvature analysis.
result Control on shape stability depends only on mean curvature.
We discuss in which sense general metric measure spaces possess a first order differential structure. Building on this, we then see that on spaces with Ricci curvature bounded from below a second order calculus can be developed, permitting to define Hessian, covariant/exterior derivatives and Ricci curvature.
A new method for training diffusion models using likelihood matching.
problem Training efficient and accurate diffusion models.
method Likelihood Matching approach, quasi-likelihood approximation, score and Hessian estimation.
result Consistent matching of first two transitional moments between diffusion steps.
SDProp improves deep neural network training efficiency by noise handling.
problem Inaccurate learning rate approximation in adaptive algorithms like RMSProp.
method SDProp uses covariance matrix preconditioning to handle noise in first order gradients.
result SDProp outperforms RMSProp and variants in various neural networks.
The paper addresses geometric analysis on non-compact Riemannian manifolds, proving Calderón-Zygmund inequalities.
problem Proving Calderón-Zygmund inequalities on non-compact Riemannian manifolds without positive injectivity radius.
method Probabilistic tools, Hessian formulas, and Bismut type representations for heat semigroups.
result The paper proves the Calderón-Zygmund inequality for 1 < p < 2 1<p<2 1 < p < 2 under a lower Ricci curvature bound, and for p > 2 p>2 p > 2 under additional curvature conditions. This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
Following P. M. H. Wilson's paper on sectional curvatures of Kahler moduli, we consider a natural Riemannian metric on a hypersurface f=1 in a real vector space, defined using the Hessian of a homogeneous polynomial f. We give examples to answer a question by Wilson about when this metric has nonpositive curvature. Als…
Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.
problem Efficient inference of online Newton methods with robustness to noise and ill-conditioning.
method Online Newton method with Hessian averaging and Nesterov's accelerated sketch-and-project solver.
result Global almost-sure convergence and asymptotic normality of the last iterate with non-asymptotic convergence guarantees.
Shampoo optimizes preconditioners for faster convergence in machine learning.
problem Improving convergence speed in machine learning optimization.
method Explicit connection between Shampoo's Kronecker product approximation and optimal matrix approximations.
result The square of Shampoo's approximation is equivalent to a single power iteration step for optimal Kronecker product approximation.
Derivative-free method solves stochastic optimization problems with noisy objectives and constraints.
problem Solving nonlinear optimization problems with stochastic objectives and deterministic constraints using only zero-order information.
method Derivative-Free Stochastic Sequential Quadratic Programming (DF-SSQP) method using simultaneous perturbation stochastic approximation (SPSA) for gradient and Hessian estimation.
result Global almost-sure convergence of the DF-SSQP method under standard assumptions, with local asymptotic normality and statistical inference.
Estimates covariance matrices for matrix-variate data via core covariance geometry.
problem Estimating covariance matrices for matrix-variate data with partial isotropy.
method Fixed-rank core covariance geometry, partial-isotropy rank-r core shrinkage estimator.
result The geometry of the space of rank-r cores is a smooth manifold.
Flat minima lead to better generalization in low-rank matrix recovery models.
problem Understanding why flat minima generalize well in overparameterized models.
method Analysis of overparameterized matrix and bilinear sensing, robust PCA, covariance matrix estimation, and neural networks with quadratic activation functions.
result Flat minima, measured by the trace of the Hessian, exactly recover the ground truth in low-rank matrix recovery models under standard statistical assumptions.
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
Study mini-batch SGD noise and its limits, proving complexity guarantees.
problem Analyzing the noise in mini-batch SGD and its impact on optimization.
method Examined the conditional covariance and diffusion limits of SGD under different sampling designs.
result Proved mean-square upper bounds and Fisher van Trees lower bounds for SGD, linking them to effective dimension and condition number.
New metric for probability measures connects physics and geometry.
problem Developing a new metric for probability measures.
method Transport Hessian metric, formulated dynamical systems.
result Connections to physics equations and mathematical models.
Investigates regularity of solutions to complex Hessian equation.
problem Regularity of solutions to complex Hessian equation.
method Analyzes solutions to Dirichlet problem with specific density condition.
result Establishes conditions for regularity of solutions.
Abstract: Survey on quadratic Hessian equations, their properties, and open problems.
problem Understanding quadratic Hessian equations and their solutions.
method Survey and review of existing research.
result Survey of entire solutions, viscosity solutions, and Hessian estimates.
Paper solves Hessian equations on Kähler manifolds.
problem Solving Hessian equations on Kähler manifolds.
method Combines elementary symmetric functions; provides sufficient and necessary condition.
result Generalizes results for Hessian and Hessian quotient equations.
New Hessian estimates for heat equations on manifolds.
problem Estimating Hessian matrices for heat-type equations on Riemannian manifolds.
method Using Bismut-Stroock Hessian formula, with explicit coefficients and delay/growth rate functions.
result Novel backward weak Harnack inequality and precise pointwise Hessian estimates for eigenfunctions.
Characterizes and describes selfsimilar Hessian manifolds with homothetic vector fields.
problem Understanding the structure and properties of selfsimilar Hessian manifolds.
method Characterization and description of selfsimilar manifolds with homothetic vector fields.
result Any selfsimilar Hessian manifold with a potential homothetic vector field is locally isomorphic to a product of radiant Hessian manifolds.
Gaussians as noise in NCE lead to exponentially bad conditioning, hindering its efficiency.
problem Exponential conditioning of Hessian in NCE with Gaussian noise.
method Using Gaussian as the noise distribution in NCE.
result Gaussian noise in NCE leads to exponentially bad conditioning of the loss Hessian.
Study calculates Hessian of Busemann function on Damek-Ricci spaces.
problem Calculating Hessian of Busemann function on Damek-Ricci spaces.
method Calculates eigenvalues of Hessian and proves positive definiteness.
result Hessian of Busemann function is positive definite.
The paper describes flat Hessian metrics on surfaces and their potentials.
problem Understanding Hessian metrics on surfaces.
method Theoretical description and explicit construction using integrable systems.
result Explicit construction of potentials for flat Hessian metrics on surfaces.
Determinantal averaging corrects inversion bias in distributed Newton's method.
problem Inverting a sum of distributed matrices is biased; local averages are incorrect.
method Reweighting local estimates of the Newton's step proportionally to the determinant of the local Hessian estimate, then averaging them.
result Determinantal averaging provides the first known asymptotically consistent distributed Newton step.
The paper shows infinitely many components in Floer Hessians space.
problem Understanding the structure of Floer Hessians.
method Proving the existence of infinitely many connected components.
result Proves infinitely many connected components in Floer Hessians space.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
The paper proves criteria for solving complex Hessian equations on projective manifolds.
problem Solving complex Hessian-type equations on projective manifolds.
method Proves Nakai-Moishezon-type criteria for equations with specific polynomial properties.
result Uniform criteria for solving complex Hessian and Hessian quotient equations.
The paper quantizes Hessian structures on R^2 using KV-algebras.
problem Quantizing Hessian structures on a 2D space.
method Deformation quantization within Koszul-Vinberg algebras.
result Established links between deformation theory and Hessian geometry.
The study provides a criterion for solving complex Hessian-type equations on projective manifolds.
problem Solving complex Hessian-type equations on projective manifolds.
method Proving Nakai-Moishezon-type criteria for these equations.
result Uniform criteria for solving these equations, including complex Hessian and Hessian quotient equations.
We prove that, in dimensions greater than 2, the generic metric is not a Hessian metric and find a curvature condition on Hessian metrics in dimensions greater than 3. In particular we prove that the forms used to define the Pontryagin classes in terms of the curvature vanish on a Hessian manifold. By contrast all anal…
Curved Frobenius manifolds link to Hessian metrics in geometry.
problem Understanding curved Frobenius manifolds and their relation to Hessian metrics.
method Analyzing the relationship between curved Frobenius structures and Hessian metrics on spaces with non-vanishing curvature.
result Consistent curved Frobenius structures on constant curvature spaces are linked to Hessian metrics.
Superintegrable systems on curved manifolds found to have Hessian structures.
problem Characterizing superintegrable systems on curved manifolds.
method Identifying and computing Hessian coordinates for superintegrable systems.
result Examples of superintegrable systems in 2D and 3D have natural Hessian coordinates.
Study classifies 3D Hessian manifolds, proving their topology.
problem Global topology of 3D Hessian manifolds.
method Proved structure and analyzed Betti numbers.
result Complete topological classification of 3D Hessian manifolds.
New rigidity results for a generalized Ricci-Hessian equation on manifolds.
problem Understanding rigidity in generalized Ricci-Hessian equations on manifolds.
method Proving new rigidity results related to a generalized Ricci-Hessian equation.
result New rigidity results for the generalized Ricci-Hessian equation on Riemannian manifolds.
Criterion for solvability of complex 2-Hessian equation on compact Kähler manifolds.
problem Solvability of complex 2-Hessian equation on compact Kähler manifolds.
method Nakai--Moishezon-type criterion associated with the complex 2-Hessian equation.
result Criterion equivalent to existence of a smooth 2-admissible representative in complex dimension three.
Establishes a concavity property for positive Hessian quotient operators.
problem Analyzing positive Hessian quotient operators on Riemannian manifolds.
method Proves a special concavity property and a Jacobi inequality.
result Proves a Jacobi inequality for symmetric tensors.
Estimates for complex Hessian equations on Hermitian manifolds.
problem Establishing estimates for solutions to complex Hessian equations.
method Using concavity inequality for complex sum-of-Hessian operators.
result Second-order estimates for admissible solutions on Hermitian manifolds.