arXiv research

This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.

Generalizes smoothness conditions for optimization methods.

We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

SGD converges to global minimum for certain non-convex functions.

Difference of convex (DC) functions cover a broad family of non-convex and possibly non-smooth and non-differentiable functions, and have wide applications in machine learning and statistics. Although deterministic algorithms for DC functions have been extensively studied, stochastic optimization that is more suitable …

Lower bounds for higher-order methods in non-convex optimization.

SGD converges to global minimum for structured non-convex functions.

Improved convergence analysis for decentralized non-convex optimization.

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

In this two-part work, we propose an algorithmic framework for solving non-convex problems whose objective function is the sum of a number of smooth component functions plus a convex (possibly non-smooth) or/and smooth (possibly non-convex) regularization function. The proposed algorithm incorporates ideas from several…

Study revisits AdaGrad convergence with relaxed noise assumptions.

New DRO algorithm finds robust models without strong assumptions.

New method for faster convergence in non-convex optimization with unbounded smoothness.

We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of conver…

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

Improved optimization technique reduces training complexity for non-convex problems.

Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The objective function is the composition of two expectations of stochastic functions, and is more challenging to optimize than vanilla stochastic o…

Advances smooth over-parameterization for solving non-smooth optimization problems.

An Euler discretization of the Langevin diffusion is known to converge to the global minimizers of certain convex and non-convex optimization problems. We show that this property holds for any suitably smooth diffusion and that different diffusions are suitable for optimizing different classes of convex and non-convex …

New sampling algorithm for non-smooth potentials.

In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…

In this paper, we study a family of non-convex and possibly non-smooth inf-projection minimization problems, where the target objective function is equal to minimization of a joint function over another variable. This problem include difference of convex (DC) functions and a family of bi-convex functions as special cas…

Paper improves stability analysis of SGD for various loss functions and data distributions.

Finding efficient and provable methods to solve non-convex optimization problems is an outstanding challenge in machine learning and optimization theory. A popular approach used to tackle non-convex problems is to use convex relaxation techniques to find a convex surrogate for the problem. Unfortunately, convex relaxat…

SGD's uncertainty quantified in non-convex learning problems.

New bounds found for optimizing non-convex functions with noisy data.

Convergence to a saddle point for convex-concave functions has been studied for decades, while recent years has seen a surge of interest in non-convex (zero-sum) smooth games, motivated by their recent wide applications. It remains an intriguing research challenge how local optimal points are defined and which algorith…

This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.

This paper studies quasar-convex functions to improve optimization methods.

New methods improve convergence in non-convex non-smooth learning problems.

This paper analyzes SGD with biased gradients for deep learning models.

In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…

The paper explores various stationarity concepts in non-smooth optimization.

Two algorithms find optimal points in decentralized optimization.

New algorithm solves non-convex, non-differentiable min-max games.

We design a stochastic algorithm to train any smooth neural network to $\varepsilon$-approximate local minima, using $O(\varepsilon^{-3.25})$ backpropagations. The best result was essentially $O(\varepsilon^{-4})$ by SGD. More broadly, it finds $\varepsilon$-approximate local minima of any smooth nonconvex function in …

Paper tackles non-convex constrained DRO with a stochastic algorithm for large-scale applications.

We investigate the theoretical limits of pipeline parallel learning of deep learning architectures, a distributed setup in which the computation is distributed per layer instead of per example. For smooth convex and non-convex objective functions, we provide matching lower and upper complexity bounds and show that a na…

We lower bound the complexity of finding $ε$-stationary points (with gradient norm at most $ε$) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (i…

Heavy Ball method speeds up finding global optima in non-convex problems.

Given a nonconvex function that is an average of $n$ smooth functions, we design stochastic first-order methods to find its approximate stationary points. The convergence of our new methods depends on the smallest (negative) eigenvalue $-σ$ of the Hessian, a parameter that describes how nonconvex the function is. Our m…

New framework for robust hypothesis testing using Sinkhorn uncertainty sets.

A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.

New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.

We consider online forecasting problems for non-convex machine learning models. Forecasting introduces several challenges such as (i) frequent updates are necessary to deal with concept drift issues since the dynamics of the environment change over time, and (ii) the state of the art models are non-convex models. We ad…

A fast method for decentralized non-convex optimization over networks.

New SGD covering technique yields dimension-independent generalization bounds.