arXiv research

Improved training of large-scale neural networks with reduced variance noise.

This paper improves NAQ method for faster convergence on Tensorflow.

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…

A common problem in training neural networks is the vanishing and/or exploding gradient problem which is more prominently seen in training of Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed for training RNNs. This paper proposes a novel adaptive stochastic Nesterov accelerated quasiNewton (…

New method solves convex optimization faster than NAG.

Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …

Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.

Develops accelerated methods for optimization using low-dimensional projected-gradient information.

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…

Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.

New method accelerates gradient descent on curved spaces.

AGNES accelerates gradient descent with noisy gradients.

Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration o…

This research accelerates sampling methods using Nesterov's Acceleration.

We present a dynamical system framework for understanding Nesterov's accelerated gradient method. In contrast to earlier work, our derivation does not rely on a vanishing step size argument. We show that Nesterov acceleration arises from discretizing an ordinary differential equation with a semi-implicit Euler integrat…

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's schem…

ANADDH uses deep learning to improve volatility risk management.

Accelerated gradient method's stability deteriorates exponentially with steps.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …

Improved optimization guarantees for deep learning models with Nesterov acceleration.

Accelerates MMLE using SVGD with Nesterov acceleration.

A new algorithm improves convergence rates for convex optimization problems.

Two new differentially private optimization algorithms derived from accelerated methods.

Poor (even random) starting points for learning/training/optimization are common in machine learning. In many settings, the method of Robbins and Monro (online stochastic gradient descent) is known to be optimal for good starting points, but may not be optimal for poor starting points -- indeed, for poor starting point…

A new transformer model accelerates training with optimization techniques.

GRU models with Adam optimizer outperform other combinations in stock market forecasting.

Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

Stacking improves deep neural network training efficiency.

Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…

Nesterov's momentum trick is famously known for accelerating gradient descent, and has been proven useful in building fast iterative algorithms. However, in the stochastic setting, counterexamples exist and prevent Nesterov's momentum from providing similar acceleration, even if the underlying problem is convex and fin…

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…

Two accelerated extragradient methods converge at $O(1/k)$ rate for co-hypomonotone inclusions.

Unified analysis of Federated Averaging and Nesterov FedAvg for linear speedup.

Proposes momentum methods for Lie groups, improving on classical algorithms.

Novel algorithm accelerates PnP methods for image deblurring and super-resolution.

There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …

Super-acceleration of gradient descent with momentum improves loss function minimization.

FedNAG improves federated learning accuracy and reduces training time.

Accelerates Riemannian gradient methods with extrapolation.

Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.

We develop a projected Nesterov's proximal-gradient (PNPG) approach for sparse signal reconstruction that combines adaptive step size with Nesterov's momentum acceleration. The objective function that we wish to minimize is the sum of a convex differentiable data-fidelity (negative log-likelihood (NLL)) term and a conv…

A fast method for training linear classifiers maximizes margins.

GD and NAG accelerate matrix factorization and neural networks.

New algorithm AG-OG optimizes separable convex-concave problems efficiently.

New algorithms accelerate solving nonlinear matrix decomposition with ReLU.

Optimizes deep learning pipelines with novel algorithms for smooth and non-smooth functions.

First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. W…