arXiv research

New tool for parallel and private stochastic convex optimization reduces query complexity.

This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.

Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…

New algorithms optimize convex functions with high-order derivatives.

Develops accelerated methods for optimization using low-dimensional projected-gradient information.

Develops accelerated fixed-point methods with delayed oracles for scientific computing.

Accelerates ERM problems with LPI-GD and improved oracle complexity.

We provide tight upper and lower bounds on the complexity of minimizing the average of $m$ convex functions using gradient and prox oracles of the component functions. We show a significant gap between the complexity of deterministic vs randomized optimization. For smooth functions, we show that accelerated gradient de…

New method accelerates steepest descent for convex optimization.

New study shows acceleration in hyperbolic spaces is impossible for strongly geodesically convex functions.

Often times, in many design problems, there is a need to select a small set of informative or representative elements from a large ground set of entities in an optimal fashion. Submodular optimization that provides for a formal way to solve such problems, has recently received significant attention from the controls co…

Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…

Gradient descent is efficient for solving feasibility problems with minimal memory and queries.

Two new differentially private optimization algorithms derived from accelerated methods.

Boosted Frank-Wolfe accelerates optimization for nonconvex problems.

Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and are the most popular variance reduction methods in recent years. Although various accelerated variants o…

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

New method shows stochastic momentum can converge quickly on optimization problems.

Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.

New method finds stationary points in bilevel optimization problems.

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…

Improved stochastic approximation method reduces residual error.

This paper proposes an accelerated proximal stochastic variance reduced gradient (ASVRG) method, in which we design a simple and effective momentum acceleration trick. Unlike most existing accelerated stochastic variance reduction methods such as Katyusha, ASVRG has only one additional variable and one momentum paramet…

We study learning properties of accelerated gradient descent methods for linear least-squares in Hilbert spaces. We analyze the implicit regularization properties of Nesterov acceleration and a variant of heavy-ball in terms of corresponding learning error bounds. Our results show that acceleration can provides faster …

Novel methods for accelerating optimization in complex bilevel and minimax problems.

MAMBA learns policies competitive with multiple conflicting oracles.

In this paper we study the limitations of parallelization in convex optimization. A convenient approach to study parallelization is through the prism of \emph{adaptivity} which is an information theoretic measure of the parallel runtime of an algorithm [BS18]. Informally, adaptivity is the number of sequential rounds a…

Proposes momentum methods for Lie groups, improving on classical algorithms.

A new optimization method, BPM, converges linearly in non-convex, non-smooth problems.

Study accelerates gradient methods in machine learning, revealing risk and stability connections.

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…

New algorithms solve convex optimization problems with limited memory.

A new transformer model accelerates training with optimization techniques.

MAPS and MAPS-SE learn from multiple suboptimal experts to improve policies efficiently.

Develops variance-reduced methods for solving generalized equations.

New algorithm optimally evaluates policies with linear approximations.

Classifier-based AI safety gates fail in self-improvement, even with advanced verification methods.

A new method for faster optimization of noisy functions.

SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization. However, SPIDER uses an accuracy-dependent stepsize that slows down the convergence in practice, and cannot handle objec…

SympFormer accelerates attention blocks using inertial dynamics on density spaces.

We introduce an efficient algorithmic framework for model selection in online learning, also known as parameter-free online learning. Departing from previous work, which has focused on highly structured function classes such as nested balls in Hilbert space, we propose a generic meta-algorithm framework that achieves o…

A quantum reinforcement learning algorithm reduces sample complexity.

There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …

Robot learns to juggle two balls from 56 minutes of experience.

Ball k-means reduces point-centroid distance computations for faster k-means clustering.

No accelerated gradient method for hyperbolic convex functions.

Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate fo…