arXiv research

Adaptive methods improve gradient descent and proximal gradient for convex optimization.

New PnP algorithm converges with relaxed proximal gradient descent.

New method finds linear relationships across multiple data blocks using proximal gradient descent with $\ell_1$ constraint.

In this paper we consider solving saddle point problems using two variants of Gradient Descent-Ascent algorithms, Extra-gradient (EG) and Optimistic Gradient Descent Ascent (OGDA) methods. We show that both of these algorithms admit a unified analysis as approximations of the classical proximal point method for solving…

Accelerates coordinate descent methods for machine learning problems.

We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's …

This paper accelerates TV regularization algorithms by unrolling proximal gradient descent.

Paper analyzes convergence of proximal algorithm in metric spaces without geodesic convexity.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

The proximal inertial gradient descent is efficient for the composite minimization and applicable for broad of machine learning problems. In this paper, we revisit the computational complexity of this algorithm and present other novel results, especially on the convergence rates of the objective function values. The no…

Asynchronous parallel optimization algorithms for solving large-scale machine learning problems have drawn significant attention from academia to industry recently. This paper proposes a novel algorithm, decoupled asynchronous proximal stochastic gradient descent (DAP-SGD), to minimize an objective function that is the…

This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional grad…

In this paper, we extend the geometric descent method recently proposed by Bubeck, Lee and Singh to tackle nonsmooth and strongly convex composite problems. We prove that our proposed algorithm, dubbed geometric proximal gradient method (GeoPG), converges with a linear rate $(1-1/\sqrtκ)$ and thus achieves the optimal …

Paper proposes distributed optimization for federated learning with theoretical guarantees.

Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm, wh…

Improved shuffling gradient methods converge faster for nonsmooth convex optimization.

We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…

A new PGA algorithm ensures stable, robust, and noise-immune solutions for non-negative inverse problems.

New algorithm proves convergence for MAP estimation with denoisers.

Federated learning approach for binary matrix factorization.

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…

Researchers study heavy-tail properties of SGD using stochastic recurrence equations.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

A new method relaxes Boolean Matrix Factorization to make it more efficient.

Stochastic Gradient Descent has been widely studied with classification accuracy as a performance measure. However, these stochastic algorithms cannot be directly used when non-decomposable pairwise performance measures are used such as Area under the ROC curve (AUC) which is a common performance metric when the classe…

Proposes a semi-implicit back propagation method for neural networks.

Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…

New Langevin Monte Carlo algorithms for sampling from nonsmooth distributions.

Guarantees convergence for black-box variational inference without modifications.

New findings show Bregman proximal algorithms can get stuck near non-stationary points.

New convergence rates found for PnP methods using MMSE denoisers.

Flow-based models generate data with improved theoretical guarantees.

Statistical analysis of algorithm unrolling for inverse problems.

Paper optimizes sparse feature selection for cancer detection using GSVP and SVM.

Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.

ProxSPS improves on SPS for regularization tasks, offering better stability and performance.

Many machine learning techniques sacrifice convenient computational structures to gain estimation robustness and modeling flexibility. However, by exploring the modeling structures, we find these "sacrifices" do not always require more computational efforts. To shed light on such a "free-lunch" phenomenon, we study the…

Neural architecture search (NAS) recently attracts much research attention because of its ability to identify better architectures than handcrafted ones. However, many NAS methods, which optimize the search process in a discrete search space, need many GPU days for convergence. Recently, DARTS, which constructs a diffe…

We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradie…

Variational Proximal Policy Optimization improves reinforcement learning from human feedback.

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

New insights show NAG and FISTA converge linearly without knowing strong convexity modulus.

Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…

New algorithm solves non-convex, non-differentiable min-max games.

Paper estimates differences in multi-attribute Gaussian graphical models using non-convex penalties.

Paper proposes a new method to separate low rank and sparse matrices without bias.