arXiv research

Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with ${O}(ε^{-1.5})$ complexity.

A new optimisation method efficiently scales Hessian-vector products for neural networks.

New Hessian-free method improves bilevel optimization for meta-learning.

Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…

Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the lim…

Meta-learning is a promising method to achieve efficient training method towards deep neural net and has been attracting increases interests in recent years. But most of the current methods are still not capable to train complex neuron net model with long-time training process. In this paper, a novel second-order meta-…

Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…

Improved SVRC algorithm reduces complexity for nonconvex optimization.

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…

Paper introduces a new optimisation method combining NG and Hessian Free for sequence training.

FD-Net predicts future dynamics from data using Hessian-Free TRCG method.

The Bethe free energy approximation is reliable when convex on a submanifold, the 'Bethe box'.

This research accelerates sampling methods using Nesterov's Acceleration.

A family of probability distributions parametrized by an open domain $Λ$ in $R^n$ defines the Fisher information matrix on this domain which is positive semi-definite. In information geometry the standard assumption has been that the Fisher information matrix tensor is positive definite defining in this way a Riemannia…

We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametriza…

Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.

Stochastic SGN method converges faster than SGD for DNNs.

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given…

Locally conformally Hessian manifolds are dense in radiant ones of rank 1.

Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, w…

New method proves dimension-free convergence for ULD in KL divergence.

Derivative-free method solves stochastic optimization problems with noisy objectives and constraints.

The paper introduces statistical and geometric structures on anti-commutable pre-Leibniz algebroids.

FDNet learns PDEs from data with fast predictions.

A parallel optimization method for convex functions using Hessian sketching and debiasing.

Paper finds local normal forms for wavefronts in flat coordinates.

Enhances influence functions for deep models without costly Hessian inversion.

Study proves structure results for homogeneous spaces supporting specific equations.

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …

New method computes affine normal directions efficiently for sparse polynomials.

New method improves online covariance estimation for SGD.

URGE improves diffusion model quality without gradients or Hessian.

The formal structure of geometrical thermodynamics is reviewed with particular emphasis on the geometry of equilibria submanifolds. On these submanifolds thermodynamic metrics are defined as the Hessian of thermodynamic potentials. Links between geometry and thermodynamics are explored for single and multiple component…

A new method solves bilevel optimization problems without Hessian inversion.

Two new covariance estimators for ROOT-SGD improve statistical inference.

Researchers explore geometric dualities in statistical manifolds.

New method improves sampling from non-convex distributions using HFHR dynamics.

DANCE method improves large-scale learning efficiency via accumulating sample strategy.

We prove a CR Obata type result that if the first positive eigenvalue of the sub-Laplacian on a compact strictly pseudoconvex pseudohermitian manifold with a divergence free pseudohermitian torsion takes the smallest possible value then, up to a homothety of the pseudohermitian structure, the manifold is the standart S…

Optimistic method adapted for faster convex-concave min-max problems.

Study on Kähler-Ricci solitons on toric manifolds, proving rigidity.

This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…

New method uses Gaussian ODE filtering to approximate likelihoods for fast ODE inverse problems.

The study proves that certain noncompact Hessian manifolds are diffeomorphic to R^n.

Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…

New metric for probability measures connects physics and geometry.

Investigates regularity of solutions to complex Hessian equation.

Abstract: Survey on quadratic Hessian equations, their properties, and open problems.