A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repe…
We compute the Hessian of quantized Ding functionals and give an elementary proof for the convexity of quantized Ding functionals along Bergman geodesics from the view point of projective geometry. We study also the asymptotic behavior of the Hessian using the Berezin-Toeplitz quantization.
Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.
problem Understanding the relationship between parameter magnitudes and Hessian curvature in deep learning models.
method Developed a matrix-free algorithm based on sketched SVDs to measure similarity between parameter masks and Hessian eigenspaces.
result Top Hessian eigenvectors tend to be concentrated around larger parameters, indicating a connection between parameter magnitudes and loss curvature.
Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the lim…
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…
RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments c…
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, w…
We study the small time asymptotics of the gradient and Hessian of the logarithm of the heat kernel at the cut locus, giving, in principle, complete expansions for both quantities. We relate the leading terms of the expansions to the structure of the cut locus, especially to conjugacy, and we provide a probabilistic in…
Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the computations and/or to implicitly implement a form of statistical regularization. In this …
HTE improves PINNs for high-dimensional, high-order PDEs by reducing computational cost and memory usage.
problem Challenges in solving high-dimensional, high-order PDEs with PINNs due to computational cost and memory constraints.
method Introduces Hutchinson Trace Estimation (HTE) to transform Hessian matrix calculations into Hessian vector products (HVP), reducing computational cost and memory usage.
result HTE significantly reduces memory consumption and computational cost, enabling faster and more efficient solution of high-dimensional and high-order PDEs.
We consider large scale empirical risk minimization (ERM) problems, where both the problem dimension and variable size is large. In these cases, most second order methods are infeasible due to the high cost in both computing the Hessian over all samples and computing its inverse in high dimensions. In this paper, we pr…
The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.
problem Characterizing complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature.
method Using a geometric flow on noncompact affine Riemannian manifolds, constructing Hessian metrics, and proving diffeomorphism.
result Complete noncompact Hessian manifolds with nonnegative Hessian sectional curvature are diffeomorphic to R^n if their tangent bundle has maximal volume growth.
We interpret the setting for a Radon transform as a submanifold of the space of generalized functions, and compute its extrinsic curvature: it is the Hessian composed with the Radon transform.