Improved Hessian-free method for neural networks reduces computational cost.
problem High computational cost and model-dependent algorithmic variations in second-order methods.
method Block-diagonal approximation of the generalized Gauss-Newton matrix, conjugate gradient updates for each block.
result Better convergence and generalization compared to original Hessian-free and Adam methods.
A new meta-learning method improves deep neural net training efficiency.
problem Efficient training of complex deep neural networks with long training processes.
method Meta-learning with Hessian-Free (MLHF) approach based on Hessian-Free optimization.
result MLHF shows good and continuous training performance in deep convolution neural nets.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…
Paper introduces a new optimisation method combining NG and Hessian Free for sequence training.
problem Overfitting and inefficiency in training DNNs with sequence criteria.
method Combines Natural Gradient and Hessian Free methods for better path traversal on parameter manifold.
result Achieves larger reductions in WER and lower WER compared to NG and HF methods.
FD-Net predicts future dynamics from data using Hessian-Free TRCG method.
problem Discovering hidden partial differential equations from data.
method Finite-difference inspired convolutional neural network with Hessian-Free TRCG method.
result FD-Net predicts future dynamics efficiently using few trainable parameters.
This research accelerates sampling methods using Nesterov's Acceleration.
problem Improving sampling efficiency in MCMC methods.
method Developed a Hessian-Free High-Resolution ODE reformulation of NAG-SC, injected noise, and discretized the diffusion process.
result Quantified acceleration beyond underdamped Langevin in W 2 W_2 W 2 distance for log-strongly-concave targets. FDNet learns PDEs from data with fast predictions.
problem Discovering complex systems behavior from data.
method Finite difference neural networks (FDNet) to learn PDEs from trajectory data.
result FDNet predicts future behavior with few trainable parameters.
Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…
DANCE method improves large-scale learning efficiency via accumulating sample strategy.
problem Efficiently solving large-scale empirical risk minimization problems.
method Distributed Accumulated Newton Conjugate gradient (DANCE) method with multistage approach.
result The method achieves satisfactory statistical accuracy with fewer passes over data.
Stochastic SGN method converges faster than SGD for DNNs.
problem Training deep neural networks efficiently.
method Stochastic generalized Gauss-Newton method using conjugate gradient and automatic differentiation.
result SGN requires fewer iterations and is more robust to hyperparameters.
A new batch optimisation framework using Natural Gradient improves DNN acoustic models.
problem Optimizing DNN acoustic models for better word error rate approximation.
method Proposes a Natural Gradient (NG) approach to sequence training, correcting the gradient based on local curvature of KL-divergence.
result The NG method converges more quickly and can be applied to any sequence discriminative training criterion.
New trust-region algorithms improve ML performance with indefinite Hessians.
problem Training machine learning models with indefinite Hessians.
method Quasi-Newton trust-region framework for indefinite Hessians.
result Proposed methods outperform traditional BFGS and Hessian-free methods.
Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given…
New multiplicative LSTM architecture outperforms RNNs in character prediction.
problem Optimizing and contrasting RNN architectures for better time series modeling.
method Hessian free optimization applied to various RNN architectures including multiplicative LSTM.
result Multiplicative LSTM hybrid architecture outperformed LSTM and multiplicative RNNs.
New method improves sampling from non-convex distributions using HFHR dynamics.
problem Sampling from non-log-concave densities with non-convex potential functions.
method Hessian-free high-resolution dynamics (HFHR) with reflection/synchronous coupling.
result HFHR dynamics converges faster than kinetic Langevin dynamics (KLD) for non-convex potentials.
New method generates continuous Pareto sets for multi-task learning.
problem Challenges in finding optimal solutions for correlated multi-task learning problems.
method Efficiently generates locally continuous Pareto sets and fronts in multi-objective optimization problems.
result Demonstrates continuous analysis of Pareto optimal solutions in machine learning problems.
Super-convergence allows neural nets to train faster with large learning rates.
problem Training neural networks too slowly.
method Training with large learning rates and one learning rate cycle.
result Neural networks can be trained an order of magnitude faster.
New method improves online covariance estimation for SGD.
problem Improving online covariance estimation for SGD.
method Proposes a de-biased covariance estimator that eliminates second-order derivatives.
result Achieves a convergence rate of n ( α − 1 ) / 2 log n n^{(α-1)/2} \sqrt{\log n} n ( α − 1 ) /2 log n , outperforming existing methods. Two new covariance estimators for ROOT-SGD improve statistical inference.
problem Uncertainty measurement for ROOT-SGD's normal distribution estimator.
method Developed two covariance estimators: plug-in and Hessian-free.
result Hessian-free estimator is asymptotically consistent and Hessian-free.
Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametriza…
This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…
ES-MAML uses Evolution Strategies for MAML, avoiding second derivative estimation.
problem Solving the MAML problem with efficient second derivative estimation.
method Applies Evolution Strategies to MAML, avoiding second derivative estimation.
result ES-MAML performs competitively and often better with fewer queries.
Stochastic second-order methods converge fast under interpolation conditions.
problem Minimizing smooth and strongly-convex functions efficiently.
method Regularized subsampled Newton method (R-SSN) and stochastic BFGS algorithms.
result R-SSN achieves global linear convergence and quadratic rate in a local neighbourhood.
Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a var…
adaQN improves training RNNs with low cost and good performance.
problem Training RNNs is computationally difficult due to vanishing/exploding gradient issues.
method Stochastic quasi-Newton algorithm with L-BFGS updating, low per-iteration cost.
result adaQN is competitive with popular RNN training algorithms on language modeling tasks.
In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases…
TrustVI is a fast second-order algorithm for black-box variational inference.
problem Efficiently optimizing variational distributions in complex models.
method Trust-region optimization with minibatch reparameterization.
result TrustVI converges faster and finds better variational distributions than alternatives.
Locally private online quantile regression method addresses privacy constraints.
problem Estimating and inferring quantile regression under local differential privacy constraints.
method Developed a finite-alphabet channel where users compute local contributions, apply randomized response, and send reports. A public decoder corrects distortion and reconstructs inputs for averaging.
result Established local privacy, decoder unbiasedness, consistency, asymptotic normality, and inference for scalar contrasts.
Enhances influence functions for deep models without costly Hessian inversion.
problem Challenges in computing Hessian matrix inverse for deep models.
method Substitutes Hessian inverse with identity matrix and extends applications.
result TracIn method performs well in measuring model utility and fairness.
AGD outperforms GD in escaping saddle points faster.
problem Finding second-order stationary points in nonconvex optimization.
method A simple variant of Nesterov's accelerated gradient descent (AGD).
result AGD finds a second-order stationary point in i l d e O ( 1 / ε 7 / 4 ) ilde{O}(1/ε^{7/4}) i l d e O ( 1/ ε 7/4 ) iterations, faster than GD's i l d e O ( 1 / ε 2 ) ilde{O}(1/ε^{2}) i l d e O ( 1/ ε 2 ) . New MAML method guarantees convergence to near-optimal solutions in nonconvex settings.
problem Convergence analysis of gradient-based MAML methods in nonconvex settings.
method Developed and analyzed MAML and FO-MAML methods, proposing Hessian-free MAML.
result MAML can find an ε-first-order stationary point after at most O(1/ε^2) iterations.
Improved SVRC algorithm reduces complexity for nonconvex optimization.
problem Finding local minima for nonconvex finite-sum optimization with improved complexity.
method Stochastic Recursive Variance-Reduced Cubic regularization (SRVRC) using recursively updated semi-stochastic gradient and Hessian estimators.
result SRVRC achieves improved gradient and Hessian complexities to find ( ε , ε ) (ε, \sqrtε) ( ε , ε ) -approximate local minimum. This paper accelerates gradient methods to find local minima in non-convex optimization.
problem Finding local minima in non-convex optimization problems.
method Polyak's Heavy Ball method and Nesterov's Accelerated Gradient method for extracting negative curvature.
result A new AG algorithm converges to second-order stationary points with improved iteration complexity.
Mini-Hes improves LFA model performance on HDI tasks with missing data.
problem Effective representation of high-dimensional, incomplete data for user behavior understanding.
method Proposes Mini-Hes, a parallelizable second-order LFA model using mini-block diagonal Hessian-free optimization.
result Mini-Hes outperforms state-of-the-art models in missing data estimation tasks on recommender system datasets.
The paper analyzes variance reduction in stochastic gradient Langevin dynamics.
problem Reducing the variance of stochastic gradient estimators in Langevin dynamics.
method Central limit theorem and Poisson equation analysis for variance characterization.
result Anti-symmetric perturbations can reduce the variance of non-reversible Langevin dynamics.
Improved iterative methods for risk parity portfolio weights.
problem Solving for portfolio weights in risk parity allocation.
method Enhanced CCD and Newton methods, including a rescaling step and improved initial guess.
result Improved CCD method is the best, three times faster with 40% fewer iterations.
Paper introduces stochastic mesh methods for efficient CVA computation.
problem Efficient computation of CVA for large systems using Monte Carlo methods.
method Introduces two stochastic mesh methods for CVA computation.
result Demonstrates the rate of convergence of the methods to real CVA values.
A new method combines Laplace and Variational Bayes for scalable inference.
problem Complex models and large datasets make exact inference infeasible.
method Low-Rank Variational Bayes Correction (VBC) using Laplace method and Variational Bayes correction in a lower dimension.
result The method ensures scalability in both model complexity and data size.
Unified framework for model explanation methods based on feature removal.
problem Unclear relationships and preferences among various model explanation methods.
method Characterizes removal-based explanations along three dimensions.
result Unified 26 existing methods, including widely used approaches.
A new method speeds up optimization for sums, simpler than others.
problem Optimizing finite sums, especially in non-smooth cases.
method A novel accelerated method based on SAGA, with one parameter.
result Achieves accelerated convergence on strongly convex smooth problems.
Unified analysis of momentum methods for deep learning.
problem Convergence analysis of stochastic momentum methods for convex and non-convex optimization.
method Developed a convergence analysis for two stochastic momentum methods.
result Unified framework revealing similarities and differences between methods.
This work reviews and evaluates methods for predicting prediction intervals in regression problems.
problem Calibration of prediction intervals in regression problems.
method Four classes of methods: Bayesian, ensemble, direct interval estimation, and conformal prediction.
result Conformal prediction can be used as a general calibration procedure.
Derives kernel PCA with Nyström method for scalability.
problem Scalability of kernel PCA.
method Nyström method for kernel PCA.
result Provides scalable alternative to full kernel PCA.
Develops a fast method for pricing American options under variance gamma model.
problem Inefficient methods for pricing American options under variance gamma model.
method Inspired by quadratic approximation method, uses machine learning on pre-calculated quantities to reduce error.
result Proposed method is efficient and accurate for practical use.
Two RBF methods solve complex financial derivatives pricing problems.
problem Pricing derivatives in models with multiple stochastic factors.
method Radial Basis Function Partition of Unity and Radial Basis Function generated Finite Differences methods.
result Both methods achieve high accuracy and are efficient for solving multi-dimensional PDEs.