arXiv research

Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.

New algorithms accelerate model-based optimization for stochastic problems.

Catalyst accelerates convex optimization methods with practical guidelines.

Paper develops accelerated APCD for nonconvex nonsmooth problems with performance guarantees.

New method accelerates Bayesian imaging using Langevin sampling.

Paper introduces a new reinforcement learning method with improved performance.

Accelerates sampling from Gibbs distributions using ARWP method.

A new optimization method, BPM, converges linearly in non-convex, non-smooth problems.

We analyze Riemannian accelerated methods using a new framework.

New algorithm speeds up solving saddle-point problems with large condition numbers.

We propose an inexact variable-metric proximal point algorithm to accelerate gradient-based optimization algorithms. The proposed scheme, called QNing can be notably applied to incremental first-order methods such as the stochastic variance-reduced gradient descent algorithm (SVRG) and other randomized incremental opti…

In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…

New algorithm accelerates single-pass SGD for generalized linear prediction.

We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framewo…

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…

We consider the problem of minimizing the sum of an average function of a large number of smooth convex components and a general, possibly non-differentiable, convex function. Although many methods have been proposed to solve this problem with the assumption that the sum is strongly convex, few methods support the non-…

Accelerates stochastic optimization for convex and strongly convex problems.

Accelerated optimization on curved geometry analyzed.

In this paper we propose a primal-dual proximal extragradient algorithm to solve the generalized Dantzig selector (GDS) estimation problem, based on a new convex-concave saddle-point (SP) reformulation. Our new formulation makes it possible to adopt recent developments in saddle-point optimization, to achieve the optim…

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

RFX accelerates and compresses Random Forests for large datasets.

In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic sol…

Accelerates coordinate descent methods for machine learning problems.

A new method combines extrapolation and line search for solving nonconvex, nonsmooth optimization problems.

Novel algorithm accelerates PnP methods for image deblurring and super-resolution.

New algorithms optimize convex functions with high-order derivatives.

Faster solution for regression and ERM problems using leverage score sampling.

New method accelerates optimization for nonconvex problems in statistics.

We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure. We analyze the runtime of the framework and obtain rates that improve state-of-the-art results for various key machine learning optimization problems including …

Unified view of accelerated and stochastic optimization methods.

ASVRG accelerates stochastic variance reduction methods with simplicity and efficiency.

Two new algorithms improve federated optimization under second-order similarity.

New methods for convex optimization with locally Lipschitz gradient, achieving faster convergence.

Inertial proximal gradient algorithm shows monotonically decreasing values.

The classical multi-set split feasibility problem seeks a point in the intersection of finitely many closed convex domain constraints, whose image under a linear mapping also lies in the intersection of finitely many closed convex range constraints. Split feasibility generalizes important inverse problems including con…

We propose a new stochastic coordinate descent method for minimizing the sum of convex functions each of which depends on a small number of coordinates only. Our method (APPROX) is simultaneously Accelerated, Parallel and PROXimal; this is the first time such a method is proposed. In the special case when the number of…

We consider a regularized least squares problem, with regularization by structured sparsity-inducing norms, which extend the usual $\ell_1$ and the group lasso penalty, by allowing the subsets to overlap. Such regularizations lead to nonsmooth problems that are difficult to optimize, and we propose in this paper a suit…

Algorithm solves robust linear regression with block Lewis weights.

New method solves convex optimization faster than NAG.

Paper solves minimax optimization gap with near-optimal algorithms.

This paper accelerates TV regularization algorithms by unrolling proximal gradient descent.

A novel BMC model with nonconvex regularizers and accelerated proximal algorithm for binary matrix completion.

Improved bounds for proximal gradient algorithms with computational errors.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…

In this paper, we propose a new algorithm to speed-up the convergence of accelerated proximal gradient (APG) methods. In order to minimize a convex function $f(\mathbf{x})$, our algorithm introduces a simple line search step after each proximal gradient step in APG so that a biconvex function $f(θ\mathbf{x})$ is minimi…

We develop a projected Nesterov's proximal-gradient (PNPG) approach for sparse signal reconstruction that combines adaptive step size with Nesterov's momentum acceleration. The objective function that we wish to minimize is the sum of a convex differentiable data-fidelity (negative log-likelihood (NLL)) term and a conv…

New algorithm solves $\ell_0$-norm constrained multilinear logistic regression for tensor data.