Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O ( ε − 1.5 ) {O}(ε^{-1.5}) O ( ε − 1.5 ) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O ( ε − 1.5 ) {O}(ε^{-1.5}) O ( ε − 1.5 ) iterations for ε ε ε -accurate stationary point. New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
Improved Hessian-free method for neural networks reduces computational cost.
problem High computational cost and model-dependent algorithmic variations in second-order methods.
method Block-diagonal approximation of the generalized Gauss-Newton matrix, conjugate gradient updates for each block.
result Better convergence and generalization compared to original Hessian-free and Adam methods.
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…
A new meta-learning method improves deep neural net training efficiency.
problem Efficient training of complex deep neural networks with long training processes.
method Meta-learning with Hessian-Free (MLHF) approach based on Hessian-Free optimization.
result MLHF shows good and continuous training performance in deep convolution neural nets.
New trust-region algorithms improve ML performance with indefinite Hessians.
problem Training machine learning models with indefinite Hessians.
method Quasi-Newton trust-region framework for indefinite Hessians.
result Proposed methods outperform traditional BFGS and Hessian-free methods.
Improved SVRC algorithm reduces complexity for nonconvex optimization.
problem Finding local minima for nonconvex finite-sum optimization with improved complexity.
method Stochastic Recursive Variance-Reduced Cubic regularization (SRVRC) using recursively updated semi-stochastic gradient and Hessian estimators.
result SRVRC achieves improved gradient and Hessian complexities to find ( ε , ε ) (ε, \sqrtε) ( ε , ε ) -approximate local minimum. Study of large batch size training using Hessian analysis and robust optimization.
problem Accuracy loss in large batch size training and robustness to adversarial perturbation.
method Hessian-based analysis and robust optimization to study large batch size training.
result Large batch size training converges to points with higher Hessian spectrum, indicating better robustness.
Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…
FD-Net predicts future dynamics from data using Hessian-Free TRCG method.
problem Discovering hidden partial differential equations from data.
method Finite-difference inspired convolutional neural network with Hessian-Free TRCG method.
result FD-Net predicts future dynamics efficiently using few trainable parameters.
This research accelerates sampling methods using Nesterov's Acceleration.
problem Improving sampling efficiency in MCMC methods.
method Developed a Hessian-Free High-Resolution ODE reformulation of NAG-SC, injected noise, and discretized the diffusion process.
result Quantified acceleration beyond underdamped Langevin in W 2 W_2 W 2 distance for log-strongly-concave targets. We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametriza…
A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Stochastic SGN method converges faster than SGD for DNNs.
problem Training deep neural networks efficiently.
method Stochastic generalized Gauss-Newton method using conjugate gradient and automatic differentiation.
result SGN requires fewer iterations and is more robust to hyperparameters.
Derivative-free method solves stochastic optimization problems with noisy objectives and constraints.
problem Solving nonlinear optimization problems with stochastic objectives and deterministic constraints using only zero-order information.
method Derivative-Free Stochastic Sequential Quadratic Programming (DF-SSQP) method using simultaneous perturbation stochastic approximation (SPSA) for gradient and Hessian estimation.
result Global almost-sure convergence of the DF-SSQP method under standard assumptions, with local asymptotic normality and statistical inference.
Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given…
A parallel optimization method for convex functions using Hessian sketching and debiasing.
problem Massively parallel optimization of convex functions with limited communication.
method Newton method with Hessian sketching and debiasing by workers, server averages descent directions.
result Approximation of Newton step with low-complexity adaptive sketching scheme.
New multiplicative LSTM architecture outperforms RNNs in character prediction.
problem Optimizing and contrasting RNN architectures for better time series modeling.
method Hessian free optimization applied to various RNN architectures including multiplicative LSTM.
result Multiplicative LSTM hybrid architecture outperformed LSTM and multiplicative RNNs.
A new method solves bilevel optimization problems without Hessian inversion.
problem Solving bilevel optimization problems in machine learning.
method Penalty method avoiding Hessian inversion.
result Asymptotically exact hypergradient and convergence under mild conditions.
Optimistic method adapted for faster convex-concave min-max problems.
problem Solving convex-concave min-max optimization problems efficiently.
method Adaptive, line search-free second-order methods combining optimistic updates and second-order information.
result Achieves optimal convergence rate without line search or backtracking.
In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …
Paper introduces a new optimisation method combining NG and Hessian Free for sequence training.
problem Overfitting and inefficiency in training DNNs with sequence criteria.
method Combines Natural Gradient and Hessian Free methods for better path traversal on parameter manifold.
result Achieves larger reductions in WER and lower WER compared to NG and HF methods.
New method generates continuous Pareto sets for multi-task learning.
problem Challenges in finding optimal solutions for correlated multi-task learning problems.
method Efficiently generates locally continuous Pareto sets and fronts in multi-objective optimization problems.
result Demonstrates continuous analysis of Pareto optimal solutions in machine learning problems.
Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…
New method computes affine normal directions efficiently for sparse polynomials.
problem Computing affine normal directions is computationally expensive in high dimensions.
method Reduces third-order tensor contraction to matrix-free formulation using log-determinant gradient.
result Scalable implementations with near-linear scaling in dimension and sparsity.
Paper finds local normal forms for wavefronts in flat coordinates.
problem Understanding local diffeomorphic types of wavefronts.
method Using connections and the metric, criteria for wavefront types are derived in affine flat coordinates.
result Local normal forms of e / m e/m e / m -wavefronts in affine flat coordinates are derived. The Bethe free energy approximation is reliable when convex on a submanifold, the 'Bethe box'.
problem Accuracy of the Bethe free energy approximation in probabilistic inference.
method Analysis of convexity and verification conditions based on the Bethe Hessian matrix.
result The Bethe approximation is mostly accurate if it is convex on a submanifold, the 'Bethe box'.
New methods for non-convex optimization using inexact Hessian approximations.
problem Optimization of non-convex functions with inexact Hessian information.
method Trust-region and cubic regularization methods with inexact Hessian approximations.
result Iteration complexity to achieve ε-approximate second-order optimality.
The paper studies efficient Hessian fitting methods for stochastic optimization.
problem Efficient Hessian fitting for stochastic optimization.
method Preconditioned Stochastic Gradient Descent (PSGD) method and Lie groups.
result Hessian fitting problem is strongly convex in certain Lie groups.
Exposes how Hessian manifold duality aids in solving optimal transport problems.
problem Solving Monge-Ampère equations and understanding mirror symmetry.
method Explains duality theory for Hessian manifolds and its application to optimal transport.
result Provides a natural setting for optimal transport and solves Monge-Ampère equations.
Optimizes quadratic bandits with tight Hessian-dependent sample complexity bounds.
problem Understanding optimal sample complexity for quadratic functions.
method Introduces energy allocation and optimal energy spectrum to prove tight lower bounds. Solves for Hessian-independent optimal algorithm.
result Proves optimal Hessian-dependent sample complexities and existence of a universally optimal algorithm.
A family of probability distributions parametrized by an open domain Λ Λ Λ in R n R^n R n defines the Fisher information matrix on this domain which is positive semi-definite. In information geometry the standard assumption has been that the Fisher information matrix tensor is positive definite defining in this way a Riemannia…
Super-convergence allows neural nets to train faster with large learning rates.
problem Training neural networks too slowly.
method Training with large learning rates and one learning rate cycle.
result Neural networks can be trained an order of magnitude faster.
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
Paper proposes a new Hessian-aware zeroth-order optimization for improving black-box adversarial attacks.
problem Improving black-box adversarial attacks on neural networks.
method Introduces a Hessian-aware zeroth-order optimization algorithm called ZO-HessAware.
result ZO-HessAware achieves improved success rates with lower query complexity.
Apollo improves nonconvex stochastic optimization efficiency.
problem Nonconvex stochastic optimization challenges.
method Adaptive parameter-wise diagonal quasi-Newton method approximating Hessian.
result Significant improvements in convergence speed and generalization over SGD and Adam.
Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.
problem Estimating the inverse Hessian for convex function minimization.
method Robbins-Monro procedure for recursive estimation of the inverse Hessian.
result Develops universal stochastic Newton methods with improved efficiency.
Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.
problem Optimizing non-convex functions like deep learning models using second-order methods.
method Nyström-approximated curvature for stochastic optimization of large-scale empirical risk minimization.
result The proposed method achieves performance competitive with state-of-the-art first-order and stochastic quasi-Newton methods.
Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.
problem Understanding the relationship between parameter magnitudes and Hessian curvature in deep learning models.
method Developed a matrix-free algorithm based on sketched SVDs to measure similarity between parameter masks and Hessian eigenspaces.
result Top Hessian eigenvectors tend to be concentrated around larger parameters, indicating a connection between parameter magnitudes and loss curvature.
New algorithms improve tensor PCA performance using Kikuchi Hessian.
problem Tensor PCA problem.
method Hierarchy of spectral methods based on Kikuchi Hessian.
result Polynomial-time algorithm matching SOS performance.
A new algorithm reduces Hessian computation for nonconvex optimization.
problem Finding local minimum in nonconvex optimization efficiently.
method Stochastic variance-reduced cubic regularization (Lite-SVRC) with constant batch size.
result Achieves lower Hessian sample complexity than existing methods.
New method uses Gaussian ODE filtering to approximate likelihoods for fast ODE inverse problems.
problem Intractable forward models in likelihood-free inference, especially for ODEs.
method Gaussian ODE filtering to construct local Gaussian likelihood approximations.
result New solvers outperform standard likelihood-free approaches on benchmark systems.
Locally conformally Hessian manifolds are dense in radiant ones of rank 1.
problem Characterizing locally conformally Hessian manifolds and their properties.
method Analyzing quotient spaces of Hessian manifolds and using statistical manifold theory.
result The set of radiant l.c.H. metrics of rank 1 is dense in all radiant l.c.H. metrics.
New algorithms estimate Hessians using random directions for faster stochastic optimization.
problem Efficiently estimating Hessians for stochastic optimization.
method Generalized Hessian estimators using random directions and noisy function measurements.
result Asymptotically unbiased estimators with lower bias for more measurements.
New method proves dimension-free convergence for ULD in KL divergence.
problem Polynomial scaling of existing convergence guarantees in high dimensions.
method Refined KL local error framework, focusing on tr(H) instead of d.
result First dimension-free KL divergence bounds for discretized ULD.
The study links model generalization to the Hessian and Lipschitz constant.
problem Understanding and improving model generalization in neural networks.
method Connecting model generalization to the Hessian and Lipschitz constant, proposing a metric and optimization algorithm.
result Model generalization is related to the Hessian and Lipschitz constant, providing a new metric and optimization algorithm.
Noise decreases the Hessian spectrum in overparameterized networks, aiding generalization.
problem Understanding why SGD leads to good generalization in overparameterized neural networks.
method Analyzing the Hessian spectrum under noise and other conditions.
result Noise decreases the trace and determinant of the Hessian spectrum in overparameterized networks.
We study Hessian fully nonlinear uniformly elliptic equations and show that the second derivatives of viscosity solutions of those equations (in 12 or more dimensions) can blow up in an interior point of the domain. We prove that the optimal interior regularity of such solutions is no more than C^{1+ε}, showing the opt…