arXiv research

Accelerates optimization in asynchronous systems with sparse updates.

Hamiltonian dynamics-based algorithms achieve deterministic and accelerated convergence for convex optimization.

Accelerates Riemannian gradient methods with extrapolation.

Develops accelerated methods for optimization using low-dimensional projected-gradient information.

New algorithms accelerate model-based optimization for stochastic problems.

FedAc accelerates Federated Averaging for distributed optimization.

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…

PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.

HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.

Accelerates policy optimization in RL with optimistic and adaptive updates.

Conditional gradients constitute a class of projection-free first-order algorithms for smooth convex optimization. As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive. However, they do not enjoy …

New algorithm accelerates optimization on Riemannian manifolds, including Wasserstein space.

In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for convex optimization, but it does not work well for nonconvex optimization typic…

Unified framework for understanding and optimizing training acceleration.

RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.

Two new differentially private optimization algorithms derived from accelerated methods.

This work accelerates gradient descent with anytime convergence guarantees.

New method accelerates steepest descent for convex optimization.

This research accelerates sampling methods using Nesterov's Acceleration.

There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …

New adaptive methods for constrained convex optimization and variational inequalities.

Accelerates MMLE using SVGD with Nesterov acceleration.

Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…

DANCE optimizes neural network and accelerator design for faster, more efficient DNN execution.

New methods optimize functions on hyperbolic and spherical spaces, matching Euclidean rates up to logarithmic factors.

New algorithm accelerates single-pass SGD for generalized linear prediction.

Accelerated gradient method's stability deteriorates exponentially with steps.

A new transformer model accelerates training with optimization techniques.

Improved NiNo networks accelerate Adam training by up to 50%.

We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…

Improved optimization guarantees for deep learning models with Nesterov acceleration.

GRU models with Adam optimizer outperform other combinations in stock market forecasting.

Accelerated optimization methods improve robustness and privacy in estimation.

Hardware-accelerated RBM solves large combinatorial problems and integer factorization.

Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …

We accelerate PMD algorithms for reinforcement learning using functional methods.

This paper studies accelerations in Q-learning algorithms. We propose an accelerated target update scheme by incorporating the historical iterates of Q functions. The idea is conceptually inspired by the momentum-based accelerated methods in the optimization theory. Conditions under which the proposed accelerated algor…

Replica exchange Langevin diffusion accelerates nonconvex optimization.

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…

Novel methods for accelerating optimization in complex bilevel and minimax problems.

Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and are the most popular variance reduction methods in recent years. Although various accelerated variants o…

We use matrix iteration theory to characterize acceleration in smooth games. We define the spectral shape of a family of games as the set containing all eigenvalues of the Jacobians of standard gradient dynamics in the family. Shapes restricted to the real line represent well-understood classes of problems, like minimi…

Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.

A new algorithm reduces the complexity of solving optimal transport problems.

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate fo…

In this paper, we introduce various mechanisms to obtain accelerated first-order stochastic optimization algorithms when the objective function is convex or strongly convex. Specifically, we extend the Catalyst approach originally designed for deterministic objectives to the stochastic setting. Given an optimization me…