A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
We analyze the Hessian spectra of large models up to 100B parameters.
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits "s…
Analyzes Hessian spectrum for neural networks near optimal learning.
Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.
Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the lim…
Wide neural networks become linear, with constant tangent kernel, due to Hessian scaling.
We solve the Dirichlet problem for -Hessian equations on compact complex manifolds with boundary, given the existence of a subsolution. Our method is based on a second order a priori estimate of the solution on the boundary with a particular gradient scale. The scale allows us to apply a blow-up argument to obtain c…
HessFormer enables distributed Hessian computation for large models.
New method for efficient sketching of gradients and Hessians.
The paper studies efficient Hessian fitting methods for stochastic optimization.
New insights into learning rates and batch sizes for neural networks using random matrix theory.
Introduces Floer functions and Floerfolds for intrinsic properties.
Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repe…
A new optimisation method efficiently scales Hessian-vector products for neural networks.
New method computes affine normal directions efficiently for sparse polynomials.
SNS accelerates Sinkhorn algorithm with sparse Newton iterations.
While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and…
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalizati…
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
We introduce a scalable measure of curvature for analyzing training dynamics of large language models.
Novel Newton method for large-scale kernel methods using random features.
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
We solve the Fu-Yau equation for arbitrary dimension and arbitrary slope . Actually we obtain at the same time a solution of the open case , an improved solution of the known case , and solutions for a family of Hessian equations which includes the Fu-Yau equation as a special case. The method is based …
Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…
The paper studies hybrid connections on Hessian manifolds and their properties.
A new hybrid Newton algorithm improves convergence in logistic regression.
New SGD algorithm finds critical points faster with second-order corrections.
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…
New pruning method captures global correlations for efficient neural network inference.
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…
Power iteration has been generalized to solve many interesting problems in machine learning and statistics. Despite its striking success, theoretical understanding of when and how such an algorithm enjoys good convergence property is limited. In this work, we introduce a new class of optimization problems called scale …
We consider self-similar solutions to mean curvature evolution of entire Lagrangian graphs. When the Hessian of the potential function has eigenvalues strictly uniformly between -1 and 1, we show that on the potential level all the shrinking solitons are quadratic polynomials while the expanding solitons are in one…
Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the computations and/or to implicitly implement a form of statistical regularization. In this …
New methods optimize functions faster with less gradient accuracy needed.
Improved graph neural network bounds using graph diffusion matrix.
We consider large scale empirical risk minimization (ERM) problems, where both the problem dimension and variable size is large. In these cases, most second order methods are infeasible due to the high cost in both computing the Hessian over all samples and computing its inverse in high dimensions. In this paper, we pr…
This work explores the importance of model weights and Hessian bias in pruning.
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly…
Deep learning algorithms often require solving a highly non-linear and nonconvex unconstrained optimization problem. Methods for solving optimization problems in large-scale machine learning, such as deep learning and deep reinforcement learning (RL), are generally restricted to the class of first-order algorithms, lik…
With the rapid development of social media sharing, people often need to manage the growing volume of multimedia data such as large scale video classification and annotation, especially to organize those videos containing human activities. Recently, manifold regularized semi-supervised learning (SSL), which explores th…
New algorithms estimate Jacobian matrices for large-scale machine learning.
NHGD solves bilevel optimization problems with reduced computational time.
New technique debiases distributed optimization, improving convergence rate.
In this paper, several modifications are introduced to the functional approximation method iterLap to reduce the approximation error, including stopping rule adjustment, proposal of new residual function, starting point selection for numerical optimisation, scaling of Hessian matrix. Illustrative examples are also prov…
Two new covariance estimators for ROOT-SGD improve statistical inference.
URGE improves diffusion model quality without gradients or Hessian.
We reparametrize ReLU NNs as splines to understand their learning dynamics.