New methods using natural gradient for structured optimization.
problem Structured optimization problems.
method Structured second-order methods via natural gradient descent.
result Efficiency demonstrated on non-convex and deep learning problems.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “second-order gradients”
Enhances gradient estimates for Hermitian Monge-Ampère equations.
problem Improving estimates for Hermitian Monge-Ampère equations.
method Improves gradient estimates using Evans-Krylov and third derivatives estimates.
result Enhanced estimates for second and third order derivatives.
New algorithms solve complex minimax problems efficiently.
problem Nonconvex-strongly concave minimax problems in machine learning.
method Gradient norm regularized trust-region (GRTR) and Levenberg-Marquardt (LMNegCur) algorithms.
result Proved iteration complexities matching best known results.
In this paper, we study stochastic non-convex optimization with non-convex random functions. Recent studies on non-convex optimization revolve around establishing second-order convergence, i.e., converging to a nearly second-order optimal stationary points. However, existing results on stochastic non-convex optimizatio…
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…
A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Informationmath.OC
Improved SVRG method using BB techniques for faster convergence.
problem Improving the convergence speed of stochastic variance reduction methods.
method Incorporates Barzilai-Borwein (BB) techniques as second-order information into SVRG.
result Proves linear convergence of the proposed method and its variants.
In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the…
Deep neural networks are usually trained with stochastic gradient descent (SGD), which minimizes objective function using very rough approximations of gradient, only averaging to the real gradient. Standard approaches like momentum or ADAM only consider a single direction, and do not try to model distance from extremum…
Variance reduction techniques like SVRG provide simple and fast algorithms for optimizing a convex finite-sum objective. For nonconvex objectives, these techniques can also find a first-order stationary point (with small gradient). However, in nonconvex optimization it is often crucial to find a second-order stationary…
Second-order methods improve differential privacy in convex optimization.
problem Improving differential privacy in convex optimization.
method Developed a private variant of the regularized cubic Newton method for strongly convex loss functions.
result Achieves quadratic convergence and optimal excess loss for strongly convex loss functions.
We analyze stochastic gradient algorithms for optimizing nonconvex problems. In particular, our goal is to find local minima (second-order stationary points) instead of just finding first-order stationary points which may be some bad unstable saddle points. We show that a simple perturbed version of stochastic recursiv…
Escaping Saddle Points for Zeroth-order Nonconvex Optimization using Estimated Gradient Descentmath.OC
Gradient descent and its variants are widely used in machine learning. However, oracle access of gradient may not be available in many applications, limiting the direct use of gradient descent. This paper proposes a method of estimating gradient to perform gradient descent, that converges to a stationary point for gene…
New SGD algorithm finds critical points faster with second-order corrections.
problem Finding critical points in non-convex optimization efficiently.
method Uses Hessian-vector products to correct momentum bias in SGD.
result Finds -critical points in time.
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton () algorithm incorporating partial second-order information.
result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
SLEDGE algorithm reduces gradient computation errors in optimization.
problem Accumulated errors in gradient estimation methods for large-scale optimization.
method Single-loop method for finite-sum nonconvex optimization without periodic gradient refresh.
result Achieves nearly optimal gradient complexity and second-order optimality.
A new algorithm for solving constrained convex optimization problems efficiently.
problem Constrained convex optimization problems requiring high accuracy solutions.
method Second-Order Conditional Gradient Sliding (SOCGS) algorithm, using projection-free methods to solve quadratic subproblems inexactly.
result Converges quadratically in primal gap after a finite number of linearly convergent iterations.
Paper improves compressed SGD to reach second-order stationary points.
problem Efficiently reaching second-order stationary points in distributed machine learning.
method Gradient compression and RandomK compressor to improve convergence to second-order stationary points.
result Compressed SGD reaches second-order stationary points with improved communication efficiency.
Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradientmath.OC
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
Paper derives estimates for complex Hessian equations on Hermitian manifolds.
problem Estimating solutions to complex Hessian equations on Hermitian manifolds.
method Derives second order estimates for solutions in a specific cone.
result Establishes second order estimates for solutions in cone.
New algorithm TURTLE outperforms MAML and meta-learner LSTM.
problem Learning new tasks quickly with limited data and compute power.
method Combining insights from MAML and meta-learner LSTM, TURTLE uses second-order gradients.
result TURTLE outperforms MAML and meta-learner LSTM in few-shot learning tasks.
The study provides interior estimates for -flows and translators in .
problem Estimating -flows and translators in .
method Proved interior gradient and second order estimates.
result Non-existence of -translators asymptotic to .
Local gradient estimates of solutions to some conformally invariant fully nonlinear equationsmath.AP
This paper concerns local gradient estimates to solutions of general conformally invariant fully nonlinear elliptic equations of second order.
PDHAMS improves sampling for discrete distributions with quadratic potential functions.
problem Sampling discrete distributions efficiently and accurately.
method Integrates a second-order approximation of the potential function and uses Gaussian integral trick.
result PDHAMS yields superior performance compared to other methods.
Improved algorithm finds second-order stationary points in non-convex optimization.
problem Minimizing non-convex objectives while preserving training data privacy.
method SpiderBoost framework with two gradient oracles: precise and less precise.
result Improved rates for finding second-order stationary points.
Second-order optimizers retain residual information after data deletion, affecting machine unlearning.
problem Residual information in second-order optimizers after data deletion.
method Comparison of first-order and second-order learners, eigendecomposition analysis.
result Second-order optimizers retain residual information, not detectable by first-order analysis.
Simple gradient descent algorithm escapes saddle points efficiently.
problem Escaping saddle points in nonconvex optimization.
method Gradient-based algorithm with polynomial iterations.
result Outputs ε-approximate second-order stationary points efficiently.
Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …
Single-site Markov Chain Monte Carlo (MCMC) is a variant of MCMC in which a single coordinate in the state space is modified in each step. Structured relational models are a good candidate for this style of inference. In the single-site context, second order methods become feasible because the typical cubic costs assoc…
Recent years have seen increased interest in performance guarantees of gradient descent algorithms for non-convex optimization. A number of works have uncovered that gradient noise plays a critical role in the ability of gradient descent recursions to efficiently escape saddle-points and reach second-order stationary p…
EvoGrad improves efficiency in meta-learning and hyperparameter optimization.
problem Efficiently compute hypergradients for larger network architectures.
method Uses evolutionary techniques to estimate hypergradients without second-order derivatives or longer computational graphs.
result Significant improvements in efficiency, enabling scaling to bigger architectures.
Optimistic method adapted for faster convex-concave min-max problems.
problem Solving convex-concave min-max optimization problems efficiently.
method Adaptive, line search-free second-order methods combining optimistic updates and second-order information.
result Achieves optimal convergence rate without line search or backtracking.
Sparse Representations of Positive Functions via First and Second-Order Pseudo-Mirror Descentstat.ML
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
New method speeds up deep learning optimization.
problem Scalable second-order optimization for deep learning.
method Second-order optimization with algorithmic and numerical improvements.
result Significant convergence and wall-clock time improvements.
Hess-MC2: Sequential Monte Carlo Squared using Hessian Information and Second Order Proposalsstat.ML
Enhances SMC² with Hessian info for more efficient posterior approximation.
problem Improving accuracy and efficiency in Bayesian inference.
method Integrates second-order information (Hessian) into SMC²'s proposal distribution.
result Second-order proposals lead to more accurate posterior approximations and better step-size selection.
Second-order guarantees for federated learning algorithms.
problem Non-convex optimization in federated learning with saddle-points as bottlenecks.
method Drawing on recent results on second-order optimality in centralized and decentralized settings, establish second-order guarantees for federated learning algorithms.
result Established second-order guarantees for federated learning algorithms.
New method learns population dynamics from snapshots, outperforming existing models.
problem Capturing periodic and other dynamical properties of population dynamics.
method Wasserstein Lagrangian Mechanics (WLM) for learning second-order dynamics from observed marginals.
result WLM outperforms existing methods across various dynamics, including vortex dynamics, embryonic development, and flocking.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…
Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposi…
A new optimizer for deep learning improves accuracy and reduces training time.
problem Training deep neural networks for classification tasks.
method Hybrid Newton/Gradient Descent (NGD) method exploiting convexity of cross-entropy loss.
result Improves validation error and provides qualitative differences in hidden layer basis functions.
Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…
New method improves online covariance estimation for SGD.
problem Improving online covariance estimation for SGD.
method Proposes a de-biased covariance estimator that eliminates second-order derivatives.
result Achieves a convergence rate of , outperforming existing methods.
First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …
New method provides tighter robustness guarantees for adversarial attacks.
problem Ensuring robustness against adversarial attacks in machine learning models.
method Developed a Second-order Smoothing (SoS) robustness certificate using Gaussian random smoothing.
result SoS certificates are tighter and provide improved robustness on high-dimensional datasets.
The paper analyzes second-order guarantees for optimization in various architectures.
problem Nonconvex optimization problems are difficult to solve, especially verifying local minima.
method Second-order guarantees for stochastic first-order optimization algorithms in different architectures.
result Gradient perturbations help algorithms distinguish and avoid undesirable stationary points.
Improved computational complexity in statistical models using second-order information.
problem Polynomial convergence of gradient descent in singular statistical models.
method Normalized Gradient Descent (NormGD) algorithm with second-order information.
result NormGD reaches final statistical radius in logarithmic iterations of .
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…
Estimates for complex equations on manifolds derived from a conjecture.
problem Estimating solutions to complex equations on Hermitian manifolds.
method Developed second order estimates for fully nonlinear elliptic equations with gradient terms.
result Derived global estimates for an equation related to Gauduchon's conjecture.