arXiv research

New PnP algorithm converges with relaxed proximal gradient descent.

Adaptive methods improve gradient descent and proximal gradient for convex optimization.

New method finds linear relationships across multiple data blocks using proximal gradient descent with $\ell_1$ constraint.

This paper accelerates TV regularization algorithms by unrolling proximal gradient descent.

Paper analyzes convergence of proximal algorithm in metric spaces without geodesic convexity.

Accelerates coordinate descent methods for machine learning problems.

NASP uses proximal gradient descent to speed up neural architecture search.

In this paper we consider solving saddle point problems using two variants of Gradient Descent-Ascent algorithms, Extra-gradient (EG) and Optimistic Gradient Descent Ascent (OGDA) methods. We show that both of these algorithms admit a unified analysis as approximations of the classical proximal point method for solving…

We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's …

Nonconvex and nonsmooth problems have recently attracted considerable attention in machine learning. However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge. Proximal coordinate descent (PCD) has been widely used for solving opti…

Unified framework for training neural networks with non-smooth, non-convex regularizers.

New Langevin Monte Carlo algorithms for sampling from nonsmooth distributions.

The proximal inertial gradient descent is efficient for the composite minimization and applicable for broad of machine learning problems. In this paper, we revisit the computational complexity of this algorithm and present other novel results, especially on the convergence rates of the objective function values. The no…

Asynchronous parallel optimization algorithms for solving large-scale machine learning problems have drawn significant attention from academia to industry recently. This paper proposes a novel algorithm, decoupled asynchronous proximal stochastic gradient descent (DAP-SGD), to minimize an objective function that is the…

New algorithm proves convergence for MAP estimation with denoisers.

Paper proposes distributed optimization for federated learning with theoretical guarantees.

In this paper, we extend the geometric descent method recently proposed by Bubeck, Lee and Singh to tackle nonsmooth and strongly convex composite problems. We prove that our proposed algorithm, dubbed geometric proximal gradient method (GeoPG), converges with a linear rate $(1-1/\sqrtκ)$ and thus achieves the optimal …

In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…

We propose and analyze a block coordinate descent proximal algorithm (BCD-prox) for simultaneous filtering and parameter estimation of ODE models. As we show on ODE systems with up to d=40 dimensions, as compared to state-of-the-art methods, BCD-prox exhibits increased robustness (to noise, parameter initialization, an…

This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional grad…

We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

New findings show Bregman proximal algorithms can get stuck near non-stationary points.

Many machine learning techniques sacrifice convenient computational structures to gain estimation robustness and modeling flexibility. However, by exploring the modeling structures, we find these "sacrifices" do not always require more computational efforts. To shed light on such a "free-lunch" phenomenon, we study the…

Federated learning approach for binary matrix factorization.

Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…

Improved greedy 2-coordinate updates for optimization problems with constraints.

A new fast algorithm solves SLOPE optimization problem.

Variational Proximal Policy Optimization improves reinforcement learning from human feedback.

A new algorithm solves signed Fréchet regression on manifolds with bounded curvature.

Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm, wh…

Guarantees convergence for black-box variational inference without modifications.

Researchers study heavy-tail properties of SGD using stochastic recurrence equations.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

New convergence rates found for PnP methods using MMSE denoisers.

A new method relaxes Boolean Matrix Factorization to make it more efficient.

Improved shuffling gradient methods converge faster for nonsmooth convex optimization.

Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.

Flow-based models generate data with improved theoretical guarantees.

Paper optimizes sparse feature selection for cancer detection using GSVP and SVM.

The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.

ProxSPS improves on SPS for regularization tasks, offering better stability and performance.

A new PGA algorithm ensures stable, robust, and noise-immune solutions for non-negative inverse problems.

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…

The $L_1$-regularized models are widely used for sparse regression or classification tasks. In this paper, we propose the orthant-wise passive descent algorithm (OPDA) for optimizing $L_1$-regularized models, as an improved substitute of proximal algorithms, which are the standard tools for optimizing the models nowada…

New algorithm improves convergence of AUC maximization.

APO optimizes neural network parameters by amortizing proximal point methods.

We propose a new stochastic coordinate descent method for minimizing the sum of convex functions each of which depends on a small number of coordinates only. Our method (APPROX) is simultaneously Accelerated, Parallel and PROXimal; this is the first time such a method is proposed. In the special case when the number of…