Accelerates Riemannian gradient methods with extrapolation.
problem Optimizing functions on manifolds efficiently.
method Extrapolating iterates in Riemannian gradient descent.
result Achieves optimal convergence rate and computational advantage.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Accelerated gradient”
AGNES accelerates gradient descent with noisy gradients.
problem Minimizing smooth convex and strongly convex functions with noisy gradients.
method Generalization of Nesterov's accelerated gradient descent algorithm for noisy conditions.
result AGNES achieves acceleration for noisy gradients with a constant of proportionality up to 1.
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
Super-acceleration of gradient descent with momentum improves loss function minimization.
problem Minimizing loss functions in machine learning.
method Extending Nesterov acceleration by using gradients at multiple steps ahead.
result Super-acceleration of the momentum algorithm is beneficial for various loss landscapes and tasks.
Accelerated gradient method's stability deteriorates exponentially with steps.
problem Algorithmic stability of Nesterov's accelerated gradient method.
method Analysis of two notions of algorithmic stability for Nesterov's accelerated gradient method.
result Stability of Nesterov's accelerated method deteriorates exponentially with the number of gradient steps.
Locally Accelerated Conditional Gradients improve convergence rates for smooth convex optimization problems.
problem Achieving optimal convergence rates for smooth convex optimization problems over polytopes.
method Locally Accelerated Conditional Gradients, coupling accelerated steps with conditional gradient steps.
result Achieves optimal accelerated local convergence for smooth strongly convex problems.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossipmath.OC
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
problem Improving the convergence rate of stochastic gradient descent and gossip algorithms.
method Introducing a continuized variant of Nesterov acceleration, which mixes variables continuously and takes gradient steps at random times.
result The continuized Nesterov acceleration achieves convergence rates similar to Nesterov's original acceleration but with random parameters.
Framework for accelerated gradient flows in Bayesian inverse problems.
problem Design efficient MCMC algorithms for Bayesian inverse problems.
method Nesterov's accelerated gradient flows in probability space, considering various information metrics.
result Proved convergence properties and proposed sampling-efficient algorithms for different metrics.
We analyze Riemannian accelerated methods using a new framework.
problem Understanding Riemannian accelerated gradient methods.
method Riemannian A-HPE framework, focusing on Euclidean A-HPE insights and metric distortion control.
result Characterization of acceleration for various Riemannian methods.
There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …
New method accelerates gradient descent on curved spaces.
problem Optimizing functions on curved Riemannian manifolds.
method Developed a novel geometric inequality to control metric distortion, enabling a Riemannian accelerated gradient method.
result Proposed the first global accelerated gradient method for Riemannian manifolds.
Acceleration in Hilbert spaces reduces computations but not accuracy.
problem Improving learning accuracy with fewer computations.
method Analysis of Nesterov acceleration and heavy-ball methods in Hilbert spaces.
result Acceleration can reduce computations but not improve accuracy with respect to gradient descent.
Hamiltonian Descent Algorithms for Optimization: Accelerated Rates via Randomized Integration Timemath.OC
HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.
problem Optimizing functions efficiently and accelerating convergence rates.
method Randomized Hamiltonian flow (RHF) with accelerated convergence rates.
result RHGD achieves accelerated convergence rates similar to Nesterov's AGD.
New adaptive and accelerated SGD methods achieve optimal convergence rates.
problem Optimizing convergence rates of stochastic gradient descent methods.
method Integrates diagonal scaling and momentum into accelerated SGD.
result Achieves optimal sampling and iteration complexity for smooth stochastic optimization.
This work accelerates gradient descent with anytime convergence guarantees.
problem Improving the convergence rate of gradient descent methods.
method Proposes a stepsize schedule for gradient descent that achieves anytime convergence rates.
result Gradient descent can achieve convergence rates of for any stopping time .
Unified analysis of conjugate gradients and accelerated methods using duality gap.
problem Minimizing convex quadratic functions efficiently.
method Approximate Duality Gap Technique to unify conjugate gradients and accelerated methods.
result Unified and self-contained proof of conjugate gradients without relying on Chebyshev polynomials.
AGBM accelerates GBM with theoretical guarantees.
problem Accumulation of errors in GBM's momentum term.
method Incorporates Nesterov's acceleration techniques and a corrected pseudo residual.
result First GBM type with theoretically-justified accelerated convergence rate.
FedAc accelerates Federated Averaging for distributed optimization.
problem Efficiently optimizing distributed machine learning models.
method Federated Accelerated Stochastic Gradient Descent (FedAc) using a potential-based perturbed iterate analysis.
result FedAc achieves faster convergence and lower communication costs than previous methods.
Study accelerates gradient methods in machine learning, revealing risk and stability connections.
problem Understanding the statistical risk of accelerated gradient methods in machine learning.
method Continuous-time analysis of Nesterov's accelerated gradient method and Polyak's heavy ball method for least squares regression.
result Connections between early stopping, stability, and curvature of loss function are revealed.
Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…
PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.
problem Locally accelerated CG requires knowledge of smoothness and strong convexity parameters.
method Parameter-Free Locally Accelerated CG (PF-LaCG) algorithm.
result PF-LaCG achieves local acceleration without requiring knowledge of smoothness and strong convexity parameters.
New methods accelerate distributed optimization in noisy networks.
problem Optimizing distributed stochastic gradient methods for noisy, connected networks.
method Developed a framework for choosing stepsize and momentum parameters, proving acceleration and providing performance bounds.
result Distributed accelerated methods achieve acceleration with optimal complexity, reducing bias and variance.
Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.
Two new differentially private optimization algorithms derived from accelerated methods.
problem Improving privacy in optimization algorithms while maintaining convergence rates.
method Polyak's heavy ball method and Nesterov's accelerated gradient method with differential privacy.
result The proposed algorithms outperform existing differentially private optimization methods.
No accelerated gradient method for hyperbolic convex functions.
problem Existence of accelerated gradient methods for geodesically convex functions on hyperbolic spaces.
method Analysis of volume growth in negatively curved spaces.
result No-go theorem for accelerated gradient methods on hyperbolic plane.
New methods accelerate gradient descent for convex and strongly convex functions.
problem Improving convergence rates of gradient-based optimization methods.
method Formulated two classes of first-order algorithms with Lyapunov analyses and Hamiltonian assisted gradient method.
result Achieved accelerated convergence rates matching Nesterov's methods in strongly and general convex settings.
A new method prunes activation gradients to speed up CNN training.
problem Challenges in accelerating CNN training using sparsity.
method Randomly prunes small activation gradients in back-propagation.
result Substantial speedups (up to 5.92x) with minimal accuracy loss.
The paper accelerates gradient flows on probability distributions using optimal control theory.
problem Optimizing probability distributions efficiently.
method Variational formulation and Hamilton's equations for accelerated gradient flows.
result The method achieves accelerated density transport from any initial distribution to a target distribution.
Accelerated optimization methods improve robustness and privacy in estimation.
problem Improving robustness and privacy in estimation methods.
method Accelerated gradient methods based on Frank-Wolfe and projected gradient descent, with tailored learning rates and Nesterov's momentum.
result Reduction in iteration complexity, leading to stronger statistical guarantees.
Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …
Accelerates policy optimization in RL with optimistic and adaptive updates.
problem Improving policy optimization methods in reinforcement learning.
method Integrates foresight into policy improvement step via optimistic and adaptive updates.
result Designs an optimistic policy gradient algorithm, adaptive via meta-gradient learning.
Gradient tree boosting is a prediction algorithm that sequentially produces a model in the form of linear combinations of decision trees, by solving an infinite-dimensional optimization problem. We combine gradient boosting and Nesterov's accelerated descent to design a new algorithm, which we call AGB (for Accelerated…
New ODEs reveal key differences in accelerated gradient methods.
problem Understanding the acceleration phenomenon in optimization algorithms.
method Developed high-resolution ODEs to distinguish between NAG-SC and Polyak's heavy-ball method.
result Identified a gradient correction term in NAG-SC not present in Polyak's method.
This paper studies an acceleration technique for incremental aggregated gradient ({\sf IAG}) method through the use of \emph{curvature} information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications. Our technique utilizes a c…
Accelerated Langevin algorithm improves MCMC sampling efficiency.
problem Improving the efficiency of MCMC sampling methods.
method Formulated gradient-based MCMC as optimization on probability measures, showing underdamped Langevin performs accelerated gradient descent.
result Accelerated rates can be achieved for nonconvex functions using the Langevin algorithm.
New method accelerates smooth games using spectral shape analysis.
problem Accelerating optimization in smooth games with complex numerical challenges.
method Matrix iteration theory and spectral shape analysis to characterize and manipulate acceleration.
result Identified a continuum of optimization strategies from convex minimization to gradient descent.
New method uses centripetal acceleration to improve GAN training.
problem Cyclic behaviors in GAN training.
method Simultaneous Centripetal Acceleration (SCA) and Alternating Centripetal Acceleration (ACA) methods.
result Gradient descent methods with SCA or ACA are linearly convergent for bilinear games.
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
Stacking improves deep neural network training efficiency.
problem Improving the efficiency of training deep neural networks.
method Proposes stacking as a form of accelerated gradient descent.
result Proves stacking provides accelerated training for certain deep linear residual networks.
A new accelerated method with simpler momentum update rules.
problem Optimizing parameters in machine learning models.
method Proposes a novel accelerated stochastic gradient method with simpler momentum update rules.
result The method outperforms Sgdm and Adam in practical problems.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
New algorithm optimally minimizes convex functions with noisy gradients.
problem Minimizing strongly convex, smooth functions with noisy gradient estimates.
method A multistage accelerated stochastic gradient method with restarts.
result Achieves optimal convergence rate in deterministic and stochastic cases.
Improved reSGLD accelerates convergence in non-convex learning problems.
problem Inefficient swaps due to noisy energy estimators in reSGLD.
method Variance reduction for noisy energy estimators, theoretical analysis, and numerical experiments.
result Exponential acceleration in convergence for non-convex learning problems.
A new gradient tree boosting framework reduces variance and accelerates performance.
problem High variance in stochastic gradient boosting.
method Combining gradient tree boosting with importance sampling and a regularizer.
result Achieves a linear convergence rate on logistic loss and 2.5x--18x acceleration on LogitBoost and LambdaMART.
Accelerated optimization on curved geometry analyzed.
problem Understanding the accelerated gradient method in curved spaces.
method Geometric interpretation of AGM on Riemannian manifold, continuous time ODE analysis.
result Convergence rate analysis of ODE for quadratic objectives.
Provably convergent acceleration in factored gradient descent with applications in matrix sensingcs.LG
We present theoretical results on the convergence of \emph{non-convex} accelerated gradient descent in matrix factorization models with -norm loss. The purpose of this work is to study the effects of acceleration in non-convex settings, where provable convergence with acceleration should not be considered a \em…