arXiv research

Survey of methods for solving smooth stochastic variational inequalities.

Stochastic approximation proves asymptotic normality for non-smooth problems.

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

We generalize stochastic smoothing for gradient estimation of non-differentiable functions.

New method improves convergence for smooth games.

In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the average of a large number of smooth component functions, and the non-smooth part is equipped with a simple proximal mapping. We propose a pr…

Paper tackles private optimization for non-smooth objectives efficiently.

Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving stochastic gradient methods named predictive local smoothness (PLS). First, we …

We propose a new algorithm---Stochastic Proximal Langevin Algorithm (SPLA)---for sampling from a log concave distribution. Our method is a generalization of the Langevin algorithm to potentials expressed as the sum of one stochastic smooth term and multiple stochastic nonsmooth terms. In each iteration, our splitting t…

We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have opti…

Introduces a new theoretical framework for exponential smoothing.

Paper improves stochastic bilevel optimization methods for highly-smooth problems.

Paper proposes ZO-SMD for MERO, achieving optimal convergence rates.

New SPS variant improves non-smooth optimization without small gradients.

Improved regret bounds for online convex optimization under stochastic and adversarial settings.

Study analyzes label smoothing in deep learning optimization.

New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.

New algorithms optimize without knowing problem parameters.

We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…

New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.

The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.

We present a new perspective on the celebrated Sinkhorn algorithm by showing that is a special case of incremental/stochastic mirror descent. In order to see this, one should simply plug Kullback-Leibler divergence in both mirror map and the objective function. Since the problem has unbounded domain, the objective func…

Generalizes smoothness conditions for optimization methods.

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipsc…

Improved spectral clustering guarantees for dynamic stochastic block models.

A new biased gradient descent method for conditional stochastic optimization.

We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…

Stochastic Gradient Descent improved for various Hilbert scales and misspecified models.

We design a stochastic algorithm to train any smooth neural network to $\varepsilon$-approximate local minima, using $O(\varepsilon^{-3.25})$ backpropagations. The best result was essentially $O(\varepsilon^{-4})$ by SGD. More broadly, it finds $\varepsilon$-approximate local minima of any smooth nonconvex function in …

Armijo line-search speeds up gradient descent for various functions.

Adapts SGD to noise and problem specifics for faster convergence.

New algorithm solves federated minimax optimization problems.

Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The objective function is the composition of two expectations of stochastic functions, and is more challenging to optimize than vanilla stochastic o…

New algorithms optimize spectral risk measures, improving interpolation between average and worst-case performance.

As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior sampling. However, SGLD typically suffers from slow convergence rate due to its large variance caused by the stochastic gradient. In order to allev…

New PG methods tackle nonconvex optimization with auto-conditioned stepsizes.

We lower bound the complexity of finding $ε$-stationary points (with gradient norm at most $ε$) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (i…

Paper analyzes convergence of stochastic methods under heavy-tailed noise.

Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…

Improved online convex optimization bounds between stochastic and adversarial settings.

In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and Hessian information of the smooth part of the objective function is available via…

New methods improve convergence in non-convex non-smooth learning problems.

Optimized method tackles convex optimization with heavy-tailed noise.

This paper optimizes attacks on stochastic bandits and proposes defenses against them.

This paper analyzes stability and generalization of Markov chain stochastic gradient methods.

We study the stochastic solution to a Cauchy problem for a degenerate parabolic equation arising from option pricing. When the diffusion coefficient of the underlying price process is locally Hölder continuous with exponent $δ\in (0, 1]$, the stochastic solution, which represents the price of a European option, is show…

This study tightens bounds on how GD and SGD generalize in smooth convex optimization problems.

New algorithms reduce complexity for solving nonconvex optimization problems with stochastic objectives and constraints.