arXiv research

Paper solves k-sparse parity problem with sign SGD, matching SQ lower bound.

Unified signSGD and gradient descent analysis for neural networks.

Efficiently performs robust and sparse kernel regression.

New algorithm eliminates sign function in PGD attacks, improving performance.

In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…

Two algorithms improve federated learning efficiency and resilience.

Interesting theoretical associations have been established by recent papers between the fields of active learning and stochastic convex optimization due to the common role of feedback in sequential querying mechanisms. In this paper, we continue this thread in two parts by exploiting these relations for the first time …

Stochastic gradient descent (SGD) is one of the most widely used optimization methods for parallel and distributed processing of large datasets. One of the key limitations of distributed SGD is the need to regularly communicate the gradients between different computation nodes. To reduce this communication bottleneck, …

Adam outperforms gradient descent on language models due to handling heavy-tailed class imbalance.

Stochastic gradient descent on manifolds improves low-rank approximation.

A new method improves stochastic gradient descent for faster and more efficient estimation.

New adaptive step-size method for convex optimization without tuning.

Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mi…

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

The paper studies stochastic gradient descent with infinite variance gradients.

Proof given for SGD convergence in a concise manner.

Unified sign-based compression for federated learning with faster convergence.

SGD reduces test error by decorrelating updates.

New dynamics for SGD in small learning rate regime.

Stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets. However, their statistical properties are not well understood, in theory. And in practice, avoiding numerical instability requires careful tuning of key parameters. Here, we introduce implicit stochastic gradien…

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…

Unified framework for distributed compressed SGD under $(L_0, L_1)$-smoothness.

Gradient descent variants improve phase retrieval accuracy.

The $L_1$-regularized models are widely used for sparse regression or classification tasks. In this paper, we propose the orthant-wise passive descent algorithm (OPDA) for optimizing $L_1$-regularized models, as an improved substitute of proximal algorithms, which are the standard tools for optimizing the models nowada…

SAGD uses Langevin algorithm for efficient gradient descent.

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

Langevin algorithms enhance training of deep neural networks for stochastic control problems.

GOLS finds activation functions affect training robustness, especially ReLU.

Stochastic gradient methods converge for training wide PINNs.

The paper analyzes stability and generalization of decentralized SGD.

Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.

Stochastic gradient descent approximates Gaussian process posteriors efficiently.

Riemannian stochastic gradient descent approximates a diffusion process called Riemannian stochastic modified flow.

We provide tight finite-time convergence bounds for gradient descent and stochastic gradient descent on quadratic functions, when the gradients are delayed and reflect iterates from $τ$ rounds ago. First, we show that without stochastic noise, delays strongly affect the attainable optimization error: In fact, the error…

Stochastic gradient descent improves Gaussian process regression.

Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…

Stochastic gradient methods can converge in expectation under heavy-tailed noise.

SGD fails to converge for deep ReLU networks with limited random initializations.

CBO interprets as SGD, leading to global convergence for nonconvex functions.

Stochastic gradient descent outperforms traditional force-directed methods.

Continuous-time SGD converges under certain conditions, useful for deep learning.

Study on test risk dynamics in learning theory with stochastic gradient flow.

In this paper, we propose a new first-order gradient-based algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in sign-based methods, e.g., SIGN-SGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we defi…

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…

Explains gradient descent methods and their convergence, focusing on simple analysis.

New stochastic gradient descent with random search directions improves efficiency and convergence.

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…

FGD reduces noisy gradient variance in SGD for neural networks.