arXiv research

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…

Paper improves rates for solving smooth minimax optimization problems.

HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.

The alternating gradient descent (AGD) is a simple but popular algorithm which has been applied to problems in optimization, machine learning, data ming, and signal processing, etc. The algorithm updates two blocks of variables in an alternating manner, in which a gradient step is taken on one block, while keeping the …

New methods resolve conflicting treatment effect estimates in health tech assessments.

Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.

Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's schem…

New dynamical system framework explains Nesterov acceleration.

Develops accelerated methods for optimization using low-dimensional projected-gradient information.

Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration o…

Improved training of large-scale neural networks with reduced variance noise.

New method accelerates gradient descent on curved spaces.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …

AGD converges in polynomial iterations to optimal matrix factorization.

AGNES accelerates gradient descent with noisy gradients.

Novel method improves training RNNs by accelerating gradient descent.

Poor (even random) starting points for learning/training/optimization are common in machine learning. In many settings, the method of Robbins and Monro (online stochastic gradient descent) is known to be optimal for good starting points, but may not be optimal for poor starting points -- indeed, for poor starting point…

Proposes a new stochastic quasi-Newton method with Nesterov's acceleration.

ANADDH uses deep learning to improve volatility risk management.

We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…

This paper improves NAQ method for faster convergence on Tensorflow.

Improved optimization guarantees for deep learning models with Nesterov acceleration.

A new algorithm improves convergence rates for convex optimization problems.

Starting with a Lie algebroid ${\cal A}$ over a space $M$ we lift its action to the canonical transformations on the affine bundle ${\cal R}$ over the cotangent bundle $T^*M$. Such lifts are classified by the first cohomology $H^1({\cal A})$. The resulting object is a Hamiltonian algebroid ${\cal A}^H$ over ${\cal R}$ …

Nesterov's extrapolation improves convergence in nonsmooth optimization.

This research accelerates sampling methods using Nesterov's Acceleration.

Unified analysis of Federated Averaging and Nesterov FedAvg for linear speedup.

We provide tight upper and lower bounds on the complexity of minimizing the average of $m$ convex functions using gradient and prox oracles of the component functions. We show a significant gap between the complexity of deterministic vs randomized optimization. For smooth functions, we show that accelerated gradient de…

GRU models with Adam optimizer outperform other combinations in stock market forecasting.

First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. W…

Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.

In this letter, we introduce a distributed Nesterov method, termed as $\mathcal{ABN}$, that does not require doubly-stochastic weight matrices. Instead, the implementation is based on a simultaneous application of both row- and column-stochastic weights that makes this method applicable to arbitrary (strongly-connected…

Accelerated gradient method's stability deteriorates exponentially with steps.

A new transformer model accelerates training with optimization techniques.

GPA improves LLM training speed by 8.71% for Llama-160M models.

We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in \cite{Nesterov2005c} for solving "fully" nonsmooth composite convex optimization problems. Our method combines both Nesterov's accelerated proximal gradient scheme and a new homotopy strategy for smoothness parameter. By an appropriat…

Unified view of gradient-based algorithms for stochastic convex composite optimization.

Accelerates MMLE using SVGD with Nesterov acceleration.

New algorithm preserves symplectic structure for faster optimization.

Two new differentially private optimization algorithms derived from accelerated methods.

New methods improve adversarial attacks' transferability to other models.

Proposes momentum methods for Lie groups, improving on classical algorithms.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

Nesterov's momentum trick is famously known for accelerating gradient descent, and has been proven useful in building fast iterative algorithms. However, in the stochastic setting, counterexamples exist and prevent Nesterov's momentum from providing similar acceleration, even if the underlying problem is convex and fin…

Stacking improves deep neural network training efficiency.

New algorithm AG-OG optimizes separable convex-concave problems efficiently.

We develop a projected Nesterov's proximal-gradient (PNPG) approach for sparse signal reconstruction that combines adaptive step size with Nesterov's momentum acceleration. The objective function that we wish to minimize is the sum of a convex differentiable data-fidelity (negative log-likelihood (NLL)) term and a conv…