arXiv research

Improved Hessian-free method for neural networks reduces computational cost.

A new meta-learning method improves deep neural net training efficiency.

New Hessian-free method improves bilevel optimization for meta-learning.

Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…

Paper introduces a new optimisation method combining NG and Hessian Free for sequence training.

FD-Net predicts future dynamics from data using Hessian-Free TRCG method.

This research accelerates sampling methods using Nesterov's Acceleration.

FDNet learns PDEs from data with fast predictions.

Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…

DANCE method improves large-scale learning efficiency via accumulating sample strategy.

Stochastic SGN method converges faster than SGD for DNNs.

A new batch optimisation framework using Natural Gradient improves DNN acoustic models.

New trust-region algorithms improve ML performance with indefinite Hessians.

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given…

New method improves sampling from non-convex distributions using HFHR dynamics.

New method generates continuous Pareto sets for multi-task learning.

Super-convergence allows neural nets to train faster with large learning rates.

New method improves online covariance estimation for SGD.

Two new covariance estimators for ROOT-SGD improve statistical inference.

Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…

We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametriza…

This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…

ES-MAML uses Evolution Strategies for MAML, avoiding second derivative estimation.

Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization…

Stochastic second-order methods converge fast under interpolation conditions.

Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a var…

adaQN improves training RNNs with low cost and good performance.

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases…

TrustVI is a fast second-order algorithm for black-box variational inference.

Locally private online quantile regression method addresses privacy constraints.

Enhances influence functions for deep models without costly Hessian inversion.

AGD outperforms GD in escaping saddle points faster.

New MAML method guarantees convergence to near-optimal solutions in nonconvex settings.

Improved SVRC algorithm reduces complexity for nonconvex optimization.

This paper accelerates gradient methods to find local minima in non-convex optimization.

Mini-Hes improves LFA model performance on HDI tasks with missing data.

The paper analyzes variance reduction in stochastic gradient Langevin dynamics.

Improved iterative methods for risk parity portfolio weights.

A new method combines Laplace and Variational Bayes for scalable inference.

Unified framework for model explanation methods based on feature removal.

A new method speeds up optimization for sums, simpler than others.

Unified analysis of momentum methods for deep learning.

This work reviews and evaluates methods for predicting prediction intervals in regression problems.

Derives kernel PCA with Nyström method for scalability.

In this paper, the author considers the numerical computation of CVA for large systems by Mote Carlo methods. He introduces two types of stochastic mesh methods for the computations of CVA. In the first method, stochastic mesh method is used to obtain the future value of the derivative contracts. In the second method, …

Develops a fast method for pricing American options under variance gamma model.

Two RBF methods solve complex financial derivatives pricing problems.