arXiv research

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

Generalizes smoothness conditions for optimization methods.

This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.

New methods improve convergence in non-convex non-smooth learning problems.

Lower bounds for higher-order methods in non-convex optimization.

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

This paper analyzes saddle points and minimax points in non-convex smooth games.

We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…

Improved optimization technique reduces training complexity for non-convex problems.

A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.

We consider online forecasting problems for non-convex machine learning models. Forecasting introduces several challenges such as (i) frequent updates are necessary to deal with concept drift issues since the dynamics of the environment change over time, and (ii) the state of the art models are non-convex models. We ad…

A new optimization method, BPM, converges linearly in non-convex, non-smooth problems.

New algorithms for differentially private optimization in convex and non-convex settings with near-optimal rates.

New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.

Step decay schedules improve convergence in non-convex optimization.

SGD converges to global minimum for certain non-convex functions.

Improved convergence analysis for decentralized non-convex optimization.

New sampling algorithm for non-smooth potentials.

This work studies the denoising of piecewise smooth graph signals that exhibit inhomogeneous levels of smoothness over a graph, where the value at each node can be vector-valued. We extend the graph trend filtering framework to denoising vector-valued graph signals with a family of non-convex regularizers, which exhibi…

New algorithm finds critical points in non-convex optimization with heavy-tailed gradients.

We investigate 1) the rate at which refined properties of the empirical risk---in particular, gradients---converge to their population counterparts in standard non-convex learning tasks, and 2) the consequences of this convergence for optimization. Our analysis follows the tradition of norm-based capacity control. We p…

Difference of convex (DC) functions cover a broad family of non-convex and possibly non-smooth and non-differentiable functions, and have wide applications in machine learning and statistics. Although deterministic algorithms for DC functions have been extensively studied, stochastic optimization that is more suitable …

Study revisits AdaGrad convergence with relaxed noise assumptions.

Study analyzes label smoothing in deep learning optimization.

New DRO algorithm finds robust models without strong assumptions.

SGD converges to global minimum for structured non-convex functions.

We prove the existence of non-smooth solutions to Special Lagrangian Equations in the non-convex case.

Advances smooth over-parameterization for solving non-smooth optimization problems.

In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…

In this two-part work, we propose an algorithmic framework for solving non-convex problems whose objective function is the sum of a number of smooth component functions plus a convex (possibly non-smooth) or/and smooth (possibly non-convex) regularization function. The proposed algorithm incorporates ideas from several…

The paper explores various stationarity concepts in non-smooth optimization.

New algorithm solves non-convex, non-differentiable min-max games.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

Two algorithms find optimal points in decentralized optimization.

In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic sol…

An Euler discretization of the Langevin diffusion is known to converge to the global minimizers of certain convex and non-convex optimization problems. We show that this property holds for any suitably smooth diffusion and that different diffusions are suitable for optimizing different classes of convex and non-convex …

This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.

Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The objective function is the composition of two expectations of stochastic functions, and is more challenging to optimize than vanilla stochastic o…

New method for faster convergence in non-convex optimization with unbounded smoothness.

SGD's uncertainty quantified in non-convex learning problems.

We lower bound the complexity of finding $ε$-stationary points (with gradient norm at most $ε$) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (i…

SMG combines shuffling and momentum for non-convex optimization.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

New algorithm for differentially private distributed optimization of smooth, non-convex problems.

We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of conver…

Proves convergence of PSGLA for sampling non-convex potentials.

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

New method smooths optimization for sparse regularization.