New averaging technique speeds up Newton method convergence.
problem Superlinear convergence of stochastic Newton methods with noisy Hessians.
method Hessian averaging to reduce noise and maintain superlinear convergence.
result Hessian averaging achieves superlinear convergence with a non-asymptotic rate.
New methods optimize functions faster with less gradient accuracy needed.
problem Optimizing complex functions with limited gradient accuracy.
method Hessian averaging and adaptive gradient sampling methods.
result Improved convergence rates for various function types.
Develops unbiased averaging methods for second order optimization in distributed systems.
problem Computing the Hessian is challenging and communication is a bottleneck in distributed optimization.
method Unbiased parameter averaging methods using sampling and sketching of the Hessian.
result Provably minimizes bias for sketched Newton directions.
In distributed optimization and distributed numerical linear algebra, we often encounter an inversion bias: if we want to compute a quantity that depends on the inverse of a sum of distributed matrices, then the sum of the inverses does not equal the inverse of the sum. An example of this occurs in distributed Newton's…
Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks wit…
We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one o…
Paper improves a method for fast global and local convergence in optimization.
problem Slow global convergence in optimization methods with noisy Hessian estimates.
method Stochastic Newton Proximal Extragradient method using HPE framework.
result Faster global linear rate and superlinear convergence in fewer iterations.
Let M be a bounded open plane domain. Let f be a continuous function on the closure of M, 3-times continuously differentiable in M, which vanish on the boundary. Polterovich and Sodin proved that the values of f cannot exceed the norm of the hessian of f, averaged over the entire domain M. In this paper we study the eq…
Survey of SDR methods for high-dimensional regression and embedding.
problem Reducing dimensionality in high-dimensional data.
method Involves both statistical and machine learning approaches, covering inverse and forward regression methods.
result Supervised Kernel Dimension Reduction is equivalent to supervised PCA.
We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified the effects of classical sketch on the strictly simpler least squares regression (LSR) problem. We establish that classical …
New findings challenge the use of flatness measures in neural networks.
problem The validity of flatness measures in assessing generalization in neural networks.
method Analysis of Hessian-based flatness norms and their relation to generalization.
result Solutions with large weights and low loss are often sharper than expected, contradicting flatness measures.
A parallel optimization method for convex functions using Hessian sketching and debiasing.
problem Massively parallel optimization of convex functions with limited communication.
method Newton method with Hessian sketching and debiasing by workers, server averages descent directions.
result Approximation of Newton step with low-complexity adaptive sketching scheme.
Gradient descent forces neural network eigenvalues to a specific threshold.
problem Understanding why gradient descent drives eigenvalues to a specific threshold.
method Introduced edge coupling, a functional on consecutive iterate pairs, to explain the trajectory towards the eigenvalue threshold.
result Gradient descent forces the Hessian eigenvalue to the threshold 2/η from arbitrary initialization. New technique debiases distributed optimization, improving convergence rate.
problem Bias in local estimates limits effectiveness of distributed second order optimization.
method Surrogate sketching and scaled regularization to eliminate bias.
result The debiased local estimates lead to faster convergence in distributed optimization.
ADAHESSIAN optimizes machine learning models with adaptive second-order methods.
problem Efficiently optimizing machine learning models with second-order methods.
method Dynamic Hessian estimation via adaptive estimates, incorporating fast approximations and moving averages.
result ADAHESSIAN achieves state-of-the-art performance across various tasks.
Study geometric properties of loss functions to understand neural network performance.
problem Understanding the geometric properties of high-dimensional loss functions to improve neural network performance.
method Combine concepts from high-dimensional probability and differential geometry to study curvature properties in lower-dimensional loss representations.
result Mean curvature in the original loss space determines if saddle points appear as minima, maxima, or flat regions.
New method for unbiased regression reduces excess risk.
problem Least squares regression with optimal solution and Hessian matrix.
method Averaged stochastic gradient descent with time-average estimator.
result Unbiased estimator with O(1/k) expected excess risk.
A new Bayesian filtering method speeds up stochastic Newton optimization.
problem Minimizing log-convex functions using stochastic methods.
method Contextualizes the problem as Bayesian inference, applying Bayesian filtering to update estimates.
result Establishes conditions for diminishing effect of older observations, akin to momentum.
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
New metric for probability measures connects physics and geometry.
problem Developing a new metric for probability measures.
method Transport Hessian metric, formulated dynamical systems.
result Connections to physics equations and mathematical models.
Investigates regularity of solutions to complex Hessian equation.
problem Regularity of solutions to complex Hessian equation.
method Analyzes solutions to Dirichlet problem with specific density condition.
result Establishes conditions for regularity of solutions.
Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.
problem Efficient inference of online Newton methods with robustness to noise and ill-conditioning.
method Online Newton method with Hessian averaging and Nesterov's accelerated sketch-and-project solver.
result Global almost-sure convergence and asymptotic normality of the last iterate with non-asymptotic convergence guarantees.
Abstract: Survey on quadratic Hessian equations, their properties, and open problems.
problem Understanding quadratic Hessian equations and their solutions.
method Survey and review of existing research.
result Survey of entire solutions, viscosity solutions, and Hessian estimates.
Paper solves Hessian equations on Kähler manifolds.
problem Solving Hessian equations on Kähler manifolds.
method Combines elementary symmetric functions; provides sufficient and necessary condition.
result Generalizes results for Hessian and Hessian quotient equations.
We present two new remarkably simple stochastic second-order methods for minimizing the average of a very large number of sufficiently smooth and strongly convex functions. The first is a stochastic variant of Newton's method (SN), and the second is a stochastic variant of cubically regularized Newton's method (SCN). W…
New Hessian estimates for heat equations on manifolds.
problem Estimating Hessian matrices for heat-type equations on Riemannian manifolds.
method Using Bismut-Stroock Hessian formula, with explicit coefficients and delay/growth rate functions.
result Novel backward weak Harnack inequality and precise pointwise Hessian estimates for eigenfunctions.
Study calculates Hessian of Busemann function on Damek-Ricci spaces.
problem Calculating Hessian of Busemann function on Damek-Ricci spaces.
method Calculates eigenvalues of Hessian and proves positive definiteness.
result Hessian of Busemann function is positive definite.
The paper shows infinitely many components in Floer Hessians space.
problem Understanding the structure of Floer Hessians.
method Proving the existence of infinitely many connected components.
result Proves infinitely many connected components in Floer Hessians space.
The paper describes flat Hessian metrics on surfaces and their potentials.
problem Understanding Hessian metrics on surfaces.
method Theoretical description and explicit construction using integrable systems.
result Explicit construction of potentials for flat Hessian metrics on surfaces.
The paper quantizes Hessian structures on R^2 using KV-algebras.
problem Quantizing Hessian structures on a 2D space.
method Deformation quantization within Koszul-Vinberg algebras.
result Established links between deformation theory and Hessian geometry.
We prove that, in dimensions greater than 2, the generic metric is not a Hessian metric and find a curvature condition on Hessian metrics in dimensions greater than 3. In particular we prove that the forms used to define the Pontryagin classes in terms of the curvature vanish on a Hessian manifold. By contrast all anal…
Superintegrable systems on curved manifolds found to have Hessian structures.
problem Characterizing superintegrable systems on curved manifolds.
method Identifying and computing Hessian coordinates for superintegrable systems.
result Examples of superintegrable systems in 2D and 3D have natural Hessian coordinates.
Curved Frobenius manifolds link to Hessian metrics in geometry.
problem Understanding curved Frobenius manifolds and their relation to Hessian metrics.
method Analyzing the relationship between curved Frobenius structures and Hessian metrics on spaces with non-vanishing curvature.
result Consistent curved Frobenius structures on constant curvature spaces are linked to Hessian metrics.
New rigidity results for a generalized Ricci-Hessian equation on manifolds.
problem Understanding rigidity in generalized Ricci-Hessian equations on manifolds.
method Proving new rigidity results related to a generalized Ricci-Hessian equation.
result New rigidity results for the generalized Ricci-Hessian equation on Riemannian manifolds.
Study classifies 3D Hessian manifolds, proving their topology.
problem Global topology of 3D Hessian manifolds.
method Proved structure and analyzed Betti numbers.
result Complete topological classification of 3D Hessian manifolds.
In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the average of a large number of smooth component functions, and the non-smooth part is equipped with a simple proximal mapping. We propose a pr…
Criterion for solvability of complex 2-Hessian equation on compact Kähler manifolds.
problem Solvability of complex 2-Hessian equation on compact Kähler manifolds.
method Nakai--Moishezon-type criterion associated with the complex 2-Hessian equation.
result Criterion equivalent to existence of a smooth 2-admissible representative in complex dimension three.
Establishes a concavity property for positive Hessian quotient operators.
problem Analyzing positive Hessian quotient operators on Riemannian manifolds.
method Proves a special concavity property and a Jacobi inequality.
result Proves a Jacobi inequality for symmetric tensors.
Estimates for complex Hessian equations on Hermitian manifolds.
problem Establishing estimates for solutions to complex Hessian equations.
method Using concavity inequality for complex sum-of-Hessian operators.
result Second-order estimates for admissible solutions on Hermitian manifolds.
Derives Hessian estimates for Lagrangian mean curvature equation.
problem Lagrangian mean curvature equation with supercritical phase and bounded second derivatives.
method Derives a priori interior Hessian estimates.
result Hessian estimates for Lagrangian mean curvature equation.
A new unbiased Hessian estimator for expectation-based objectives.
problem Estimating Hessian for objectives with non-reparameterizable nodes.
method GO Hessian estimator for expectation-based objectives.
result GO Hessian provides unbiased and low-variance estimation of Hessian.
Constructs homogeneous Kähler structures on tangent bundles of Hessian manifolds.
problem Creating Kähler structures on tangent bundles of Hessian manifolds.
method Endowing Hessian manifolds with Kähler structures using group actions and homothetic vector fields.
result Homogeneous conformally Kähler structures on tangent bundles of selfsimilar Hessian manifolds.
FA-HMC improves Bayesian federated learning with rigorous guarantees.
problem Parameter estimation and uncertainty quantification in non-iid distributed data.
method Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC) with convergence guarantees.
result FA-HMC achieves better convergence and communication efficiency than existing methods.
Paper proves inequalities on Hermitian manifolds with applications to bounded solutions.
problem Establishing mixed Hessian inequalities on Hermitian manifolds.
method Weak convergence theorem of complex Hessian operators and general mixed Hessian inequality.
result Existence of bounded solutions of complex Hessian equations.
A selfsimiar manifold is a Riemannian manifold (M,g) endowed with a homothetic vector field ξ. We characterize global selfsimilar manifolds and describe the structure of local selfsimilar manifolds. We prove that any selfsimilar manifold with a potential homothetic vector field is a conical Riemannian ma…
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
New method approximates curvature from symmetries in deep networks.
problem Hard to approximate curvature in large deep networks.
method Analytically averaging over group actions that leave the loss invariant to construct structured Hessian approximations.
result Structured Hessian approximations from single gradients can be estimated, stored, and inverted.