arXiv research

Improved analysis of accelerated noisy power method for PCA.

Hamiltonian dynamics-based algorithms achieve deterministic and accelerated convergence for convex optimization.

This work accelerates gradient descent with anytime convergence guarantees.

New method accelerates convergence for entropy-regularized reinforcement learning problems.

HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.

Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.

This paper studies accelerations in Q-learning algorithms. We propose an accelerated target update scheme by incorporating the historical iterates of Q functions. The idea is conceptually inspired by the momentum-based accelerated methods in the optimization theory. Conditions under which the proposed accelerated algor…

Accelerates TD learning for long-horizon reinforcement learning problems.

We present theoretical results on the convergence of \emph{non-convex} accelerated gradient descent in matrix factorization models with $\ell_2$-norm loss. The purpose of this work is to study the effects of acceleration in non-convex settings, where provable convergence with acceleration should not be considered a \em…

The extragradient method accelerates convergence in complex game dynamics.

Anderson acceleration (or Anderson mixing) is an efficient acceleration method for fixed point iterations $x_{t+1}=G(x_t)$, e.g., gradient descent can be viewed as iteratively applying the operation $G(x) \triangleq x-α\nabla f(x)$. It is known that Anderson acceleration is quite efficient in practice and can be viewed…

New algorithms accelerate SVGD convergence using deep unfolding.

Improved convergence of fixed-point methods using windowed Anderson acceleration.

Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration o…

Paper accelerates Bayesian few-shot classification using mirror descent.

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…

New study on No-U-Turn Sampler for accelerated mixing in Hamiltonian Monte Carlo.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

New methods accelerate gradient descent for convex and strongly convex functions.

Chebyshev steps improve convergence in deep-unfolded gradient descent.

A new algorithm improves convergence rates for convex optimization problems.

RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.

Momentum is a popular technique to accelerate the convergence in practical training, and its impact on convergence guarantee has been well-studied for first-order algorithms. However, such a successful acceleration technique has not yet been proposed for second-order algorithms in nonconvex optimization.In this paper, …

Improved reSGLD accelerates convergence in non-convex learning problems.

Accelerates Riemannian gradient methods with extrapolation.

New method accelerates optimization in fixed time, improving convergence rates.

PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimi…

Novel geometry-informed irreversible perturbation accelerates Langevin dynamics convergence.

This research analyzes and accelerates score-based diffusion models using discretization and Hessian information.

Two accelerated extragradient methods converge at $O(1/k)$ rate for co-hypomonotone inclusions.

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update w…

In this paper, we study the stochastic gradient descent (SGD) method for the nonconvex nonsmooth optimization, and propose an accelerated SGD method by combining the variance reduction technique with Nesterov's extrapolation technique. Moreover, based on the local error bound condition, we establish the linear converge…

New method accelerates Bayesian imaging using Langevin sampling.

New optimizer G-AdaGrad improves upon AdaGrad for non-convex machine learning problems.

Conditional gradients constitute a class of projection-free first-order algorithms for smooth convex optimization. As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive. However, they do not enjoy …

Recently, many variance reduced stochastic alternating direction method of multipliers (ADMM) methods (e.g.\ SAG-ADMM, SDCA-ADMM and SVRG-ADMM) have made exciting progress such as linear convergence rates for strongly convex problems. However, the best known convergence rate for general convex problems is O(1/T) as opp…

Accelerated gradient method tackles nonconvex penalties in sparse learning.

Proposes EDM algorithm to accelerate model training in distributed networks.

Paper accelerates diffusion models, improving sampling speed.

Improved SGD for non-strongly-convex regression with faster convergence.

Kaczmarz++ accelerates convergence for ill-conditioned systems.

This paper analyzes and guarantees convergence of prior-guided ZO algorithms.

Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…

Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…

Two new differentially private optimization algorithms derived from accelerated methods.

Large stepsizes can accelerate gradient descent for logistic regression.

In this paper, we introduce various mechanisms to obtain accelerated first-order stochastic optimization algorithms when the objective function is convex or strongly convex. Specifically, we extend the Catalyst approach originally designed for deterministic objectives to the stochastic setting. Given an optimization me…