arXiv research

FedAc accelerates Federated Averaging for distributed optimization.

HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.

We accelerate PMD algorithms for reinforcement learning using functional methods.

New methods accelerate gradient descent for convex and strongly convex functions.

We consider gradient descent with `momentum', a widely used method for loss function minimization in machine learning. This method is often used with `Nesterov acceleration', meaning that the gradient is evaluated not at the current position in parameter space, but at the estimated position after one step. In this work…

This paper studies accelerations in Q-learning algorithms. We propose an accelerated target update scheme by incorporating the historical iterates of Q functions. The idea is conceptually inspired by the momentum-based accelerated methods in the optimization theory. Conditions under which the proposed accelerated algor…

We formulate gradient-based Markov chain Monte Carlo (MCMC) sampling as optimization on the space of probability measures, with Kullback-Leibler (KL) divergence as the objective functional. We show that an underdamped form of the Langevin algorithm performs accelerated gradient descent in this metric. To characterize t…

No accelerated gradient method for hyperbolic convex functions.

New methods optimize functions on hyperbolic and spherical spaces, matching Euclidean rates up to logarithmic factors.

New study shows acceleration in hyperbolic spaces is impossible for strongly geodesically convex functions.

New algorithms accelerate solving nonlinear matrix decomposition with ReLU.

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…

PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.

Paper proposes a new method to speed up diffusion models.

Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.

This research accelerates sampling methods using Nesterov's Acceleration.

Accelerates MMLE using SVGD with Nesterov acceleration.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

Novel methods for accelerating optimization in complex bilevel and minimax problems.

Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …

Study accelerates gradient methods in machine learning, revealing risk and stability connections.

Derives PF-ODE for infinite-dimensional functions, improving function generation tasks.

Accelerated method finds critical points faster on manifolds.

New method accelerates steepest descent for convex optimization.

New algorithm accelerates optimization on Riemannian manifolds, including Wasserstein space.

New adaptive methods for constrained convex optimization and variational inequalities.

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…

We formulate and study a general family of (continuous-time) stochastic dynamics for accelerated first-order minimization of smooth convex functions. Building on an averaging formulation of accelerated mirror descent, we propose a stochastic variant in which the gradient is contaminated by noise, and study the resultin…

We present theoretical results on the convergence of \emph{non-convex} accelerated gradient descent in matrix factorization models with $\ell_2$-norm loss. The purpose of this work is to study the effects of acceleration in non-convex settings, where provable convergence with acceleration should not be considered a \em…

ColaBO accelerates optimization with user beliefs.

GPU-accelerated particle methods outperform neural samplers in LFT benchmarks.

Anderson acceleration (or Anderson mixing) is an efficient acceleration method for fixed point iterations $x_{t+1}=G(x_t)$, e.g., gradient descent can be viewed as iteratively applying the operation $G(x) \triangleq x-α\nabla f(x)$. It is known that Anderson acceleration is quite efficient in practice and can be viewed…

This paper presents a methodology and numerical algorithms for constructing accelerated gradient flows on the space of probability distributions. In particular, we extend the recent variational formulation of accelerated gradient methods in (wibisono, et. al. 2016) from vector valued variables to probability distributi…

We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. The approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary p…

SympFormer accelerates attention blocks using inertial dynamics on density spaces.

DeltaBO accelerates Bayesian optimization with theoretical guarantees.

A new family of momentum coefficients improves the convergence rate of accelerated algorithms.

This paper analyzes and improves monotonic accelerated algorithms like M-NAG and M-FISTA.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimi…

We consider the problem of minimizing the sum of an average function of a large number of smooth convex components and a general, possibly non-differentiable, convex function. Although many methods have been proposed to solve this problem with the assumption that the sum is strongly convex, few methods support the non-…

We provide tight upper and lower bounds on the complexity of minimizing the average of $m$ convex functions using gradient and prox oracles of the component functions. We show a significant gap between the complexity of deterministic vs randomized optimization. For smooth functions, we show that accelerated gradient de…

The paper evaluates functions of stable Lévy processes and their extrema efficiently.

Accelerated gradient method tackles nonconvex penalties in sparse learning.

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…

In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits the structure of common nonsmooth loss functions to achieve optimal convergence ra…

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…

Paper accelerates diffusion models, improving sampling speed.