arXiv research

Paper introduces a new multi-kernel algorithm for better gradient approximation.

This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.

Overview of non-stochastic-gradient SA algorithms in signal processing and ML.

SGBD algorithm improves robustness in Bayesian sampling.

Stochastic approximation algorithms show exponential progress bounds.

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general noncon…

SAGD uses Langevin algorithm for efficient gradient descent.

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment po…

Unified framework connects stochastic optimization to Bayesian inference.

A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.

We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood math…

Stochastic gradient algorithms estimate the gradient based on only one or a few samples and enjoy low computational cost per iteration. They have been widely used in large-scale optimization problems. However, stochastic gradient algorithms are usually slow to converge and achieve sub-linear convergence rates, due to t…

Langevin algorithms enhance training of deep neural networks for stochastic control problems.

The paper studies stochastic gradient descent with infinite variance gradients.

A new tamed stochastic gradient Hamiltonian Monte Carlo algorithm for superlinearly growing stochastic gradients.

Optimal algorithms for online convex optimization with missing sub-gradient observations.

A new method for stochastic optimization using virtual gradients.

New method for zeroth-order stochastic gradient algorithms provides confidence intervals.

We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference …

Unified algorithm for stochastic optimization with time-varying momentum converges under general conditions.

Improved SVRC algorithm reduces complexity for nonconvex optimization.

Gradient descent variants improve phase retrieval accuracy.

An algorithm is proposed for solving stochastic and finite sum minimization problems. Based on a trust region methodology, the algorithm employs normalized steps, at least as long as the norms of the stochastic gradient estimates are within a specified interval. The complete algorithm---which dynamically chooses whethe…

STORM-PG uses momentum for faster policy gradient updates.

In this paper, we propose an implicit gradient descent algorithm for the classic $k$-means problem. The implicit gradient step or backward Euler is solved via stochastic fixed-point iteration, in which we randomly sample a mini-batch gradient in every iteration. It is the average of the fixed-point trajectory that is c…

New convergence guarantees for learning with unknown nuisance parameters.

Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to important successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new ad…

We introduce Kalman Gradient Descent, a stochastic optimization algorithm that uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent by filtering the gradient estimates. We present both a theoretical analysis of convergence in a non-convex setting and experimental results which dem…

A new algorithm SRG-DQN reduces variance in deep Q-learning.

New oracles improve stochastic optimization with noisy or biased measurements.

Unified view of gradient-based algorithms for stochastic convex composite optimization.

A new method improves stochastic gradient descent for faster and more efficient estimation.

This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these alg…

Stochastic gradient descent on manifolds improves low-rank approximation.

RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments c…

Adaptive algorithm improves convergence rate of Langevin dynamics.

New SAGA algorithm with decreasing step for stochastic optimization.

The paper analyzes SW-SGD for MSE in biased and variance-reduced gradient estimators.

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…

New method efficiently computes gradients for stochastic differential equations.

DMGD algorithm solves large-scale ML problems in costly sampling settings.

A method for estimating the median of gradients in stochastic optimization.

Stochastic multi-gradient method tackles MOO problems with uncertain data.

Improved convergence for nonconvex optimization with dependent data.

New stochastic gradient descent with random search directions improves efficiency and convergence.

Stochastic gradient methods converge for training wide PINNs.

Paper approximates risk measures using SGD with Langevin dynamics.

This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only $\mathcal{\tilde{O}}(ε^{-3.5})$ stochastic gradien…