arXiv research

New optimizer G-AdaGrad improves upon AdaGrad for non-convex machine learning problems.

Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.

AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.

This study analyzes AdaGrad's stability and convergence in non-convex optimization.

Adaptive stochastic gradient methods such as AdaGrad have gained popularity in particular for training deep neural networks. The most commonly used and studied variant maintains a diagonal matrix approximation to second order information by accumulating past gradients which are used to tune the step size adaptively. In…

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsi…

We study the implicit bias of AdaGrad on separable linear classification problems. We show that AdaGrad converges to a direction that can be characterized as the solution of a quadratic optimization problem with the same feasible set as the hard SVM problem. We also give a discussion about how different choices of the …

AdaGrad converges under heavy-tailed noise without extra operations.

Proposes a new adaptive gradient method based on gradient differences.

AdaGrad outperforms SGD in non-convex optimization problems by a factor of d.

Improved private AdaGrad achieves faster convergence rates for convex functions.

Efficient optimization method reduces Full AdaGrad complexity.

Study revisits AdaGrad convergence with relaxed noise assumptions.

GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.

Improved SGD with AdaGrad stepsizes adapts to unknown parameters and unbounded gradients.

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…

Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…

We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound wh…

Private adaptive methods improve on traditional SGD for convex optimization.

AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.

This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.

We prove that the norm version of the adaptive stochastic gradient method (AdaGrad-Norm) achieves a linear convergence rate for a subset of either strongly convex functions or non-convex functions that satisfy the Polyak Lojasiewicz (PL) inequality. The paper introduces the notion of Restricted Uniform Inequality of Gr…

This work provides formal guarantees for heuristic optimization methods in machine learning.

AdaSVRG combines adaptive gradient with SVRG for robust optimization.

The adaptive gradient online learning method known as AdaGrad has seen widespread use in the machine learning community in stochastic and adversarial online learning problems and more recently in deep learning methods. The method's full-matrix incarnation offers much better theoretical guarantees and potentially better…

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show $\sqrt{T}$-…

This paper proves AdaGrad and Adam converge linearly under PL inequality.

A parameter-free PGD algorithm for convex optimization.

Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSPr…

Advanced optimization algorithms such as Newton method and AdaGrad benefit from second order derivative or second order statistics to achieve better descent directions and faster convergence rates. At their heart, such algorithms need to compute the inverse or inverse square root of a matrix whose size is quadratic of …

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling t…

DoWG optimizer automatically adapts to convex and nonsmooth problems without tuning.

Paper analyzes high probability convergence of adaptive SGD with momentum.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drasti…

New adaptive methods for constrained convex optimization and variational inequalities.

New adaptive importance samplers improve stability and accuracy.

Given any increasing sequence of norms $\|\cdot\|_0,\dots,\|\cdot\|_{T-1}$, we provide an online convex optimization algorithm that outputs points $w_t$ in some domain $W$ in response to convex losses $\ell_t:W\to \mathbb{R}$ that guarantees regret $R_T(u)=\sum_{t=1}^T \ell_t(w_t)-\ell_t(u)\le \tilde O\left(\|u\|_{T-1}…

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in o…

This study explains how different training methods affect the minimizer of neural networks.

Prodigy estimates learning rate without tuning, improving convergence.

A new method for uncertainty estimation in neural networks using existing optimization steps.

This paper improves Adam's performance in machine learning tasks.

New analysis shows GMD can converge linearly under PL-like conditions.

Adaptive gradient methods converge faster with over-parameterization and line-search.

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with …

Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …

AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.