arXiv research

Proof of convergence for multi-objective optimization using inverse reinforcement learning.

We propose a fair principal component analysis method that balances reconstruction error and subgroup fairness.

Nesterov's extrapolation improves convergence in nonsmooth optimization.

A parameter-free PGD algorithm for convex optimization.

We propose graph-dependent implicit regularisation strategies for distributed stochastic subgradient descent (Distributed SGD) for convex problems in multi-agent learning. Under the standard assumptions of convexity, Lipschitz continuity, and smoothness, we establish statistical learning rates that retain, up to logari…

A distributed subgradient method tackles non-convex optimization problems in networks.

New method bounds stochastic subgradient methods with heavy-tailed noise.

Inexact subgradient methods work well for semialgebraic functions with additive errors.

SGD avoids critical points on weakly convex functions.

Stochastic subgradient descent avoids critical points in definable functions.

We consider the task of training classifiers without labels. We propose a weakly supervised method---adversarial label learning---that trains classifiers to perform well against an adversary that chooses labels for training data. The weak supervision constrains what labels the adversary can choose. The method therefore…

In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to…

Exact method found for estimating ILP weights from data.

Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…

We describe novel subgradient methods for a broad class of matrix optimization problems involving nuclear norm regularization. Unlike existing approaches, our method executes very cheap iterations by combining low-rank stochastic subgradients with efficient incremental SVD updates, made possible by highly optimized and…

This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can provably recover orthogonal dictionaries on a natural nonsmooth, nonconvex $\ell_1$ minimization formulation of the problem, under mi…

New method for optimization on Hadamard manifolds with curvature-independent guarantees.

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

We consider the problem of unconstrained online convex optimization (OCO) with sub-exponential noise, a strictly more general problem than the standard OCO. In this setting, the learner receives a subgradient of the loss functions corrupted by sub-exponential noise and strives to achieve optimal regret guarantee, witho…

Convex optimization method recovers low-rank matrices from rank-one projections efficiently.

SGD converges to critical points of normalized margin in late-stage training for homogeneous neural networks.

In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS$_\varepsilon$) and least squares boosting (LS-Boost($…

In this paper, we propose a convergent parallel best-response algorithm with the exact line search for the nondifferentiable nonconvex sparsity-regularized rank minimization problem. On the one hand, it exhibits a faster convergence than subgradient algorithms and block coordinate descent algorithms. On the other hand,…

Improved method reduces projection calls for nonsmooth convex optimization.

Stochastic algorithm achieves sublinear convergence for bi-objective optimization.

Optimizes distributions robustly with Sinkhorn distance.

We generalize stochastic subgradient descent methods to situations in which we do not receive independent samples from the distribution over which we optimize, but instead receive samples that are coupled over time. We show that as long as the source of randomness is suitably ergodic---it converges quickly enough to a …

New algorithm reduces privacy loss in SGD without learning rate tuning.

We propose an algorithm, semismooth Newton coordinate descent (SNCD), for the elastic-net penalized Huber loss regression and quantile regression in high dimensional settings. Unlike existing coordinate descent type algorithms, the SNCD updates each regression coefficient and its corresponding subgradient simultaneousl…

In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits the structure of common nonsmooth loss functions to achieve optimal convergence ra…

CD methods tackle nonconvex optimization with three terms, achieving critical points.

Paper introduces SGA for barycenter optimization in optimal transport.

The paper guarantees global stability for stochastic subgradient methods in nonsmooth nonconvex optimization.

Subgradient algorithm achieves optimal regret for both adversarial and i.i.d. costs on the simplex.

Paper proposes fast, robust methods for low-rank matrix recovery.

Most traditional online learning algorithms are based on variants of mirror descent or follow-the-leader. In this paper, we present an online algorithm based on a completely different approach, tailored for transductive settings, which combines "random playout" and randomized rounding of loss subgradients. As an applic…

New insights show NAG and FISTA converge linearly without knowing strong convexity modulus.

This paper certifies cluster assignments from sum-of-norms clustering algorithms.

Improved subgradient method tackles ill-conditioned composite optimization problems.

New algorithms accelerate model-based optimization for stochastic problems.

Density ratio estimation is a vital tool in both machine learning and statistical community. However, due to the unbounded nature of density ratio, the estimation procedure can be vulnerable to corrupted data points, which often pushes the estimated ratio toward infinity. In this paper, we present a robust estimator wh…

New Max-Plus neural network exploits subgradient sparsity for efficient training.

Minimax optimization plays a key role in adversarial training of machine learning algorithms, such as learning generative models, domain adaptation, privacy preservation, and robust learning. In this paper, we demonstrate the failure of alternating gradient descent in minimax optimization problems due to the discontinu…

The paper derives subgradient estimates for a specific nonlinear subparabolic equation on pseudo-Hermitian manifolds.

Study shows double descent curve in high-dimensional linear regression with random projections.

Gaussian Markov random fields (GMRFs) are useful in a broad range of applications. In this paper we tackle the problem of learning a sparse GMRF in a high-dimensional space. Our approach uses the l1-norm as a regularization on the inverse covariance matrix. We utilize a novel projected gradient method, which is faster …

Unified Lagrangian-based methods for nonsmooth nonconvex optimization.

New algorithms achieve high-probability parameter-free regret in online convex optimization with heavy-tailed data.