New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Adaptive methods improve gradient descent and proximal gradient for convex optimization.
problem Improving efficiency of gradient descent and proximal gradient methods.
method Adaptive versions of GD and ProxGD using local curvature information.
result Proved convergence with local Lipschitz gradient assumptions.
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
This paper proves AdaGrad and Adam converge linearly under PL inequality.
problem Understanding the convergence of adaptive gradient methods.
method Unified approach proving AdaGrad and Adam converge linearly under PL inequality.
result AdaGrad and Adam converge linearly when the cost function is smooth and satisfies PL inequality.
New methods using natural gradient for structured optimization.
problem Structured optimization problems.
method Structured second-order methods via natural gradient descent.
result Efficiency demonstrated on non-convex and deep learning problems.
Direct proof shows adaptive gradient descent converges near-linearly for convex functions.
problem Proving near-linear convergence of adaptive gradient descent for convex functions.
method Direct Lyapunov-based argument for convex functions with unique minimizer.
result Direct proof of near-linear convergence for convex functions.
GeoAdaLer enhances geometric understanding of Adam for stochastic optimization.
problem Understanding geometric principles behind Adam's success in stochastic optimization.
method Introduces GeoAdaLer, an adaptive learning method based on geometric properties.
result Extends interpretability and effectiveness in complex optimization scenarios.
Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…
We introduce Kalman Gradient Descent, a stochastic optimization algorithm that uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent by filtering the gradient estimates. We present both a theoretical analysis of convergence in a non-convex setting and experimental results which dem…
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…
Proposes VSGD optimizer combining probabilistic and gradient-based methods.
problem Uncertainty modeling in deep neural networks.
method Combines probabilistic and gradient-based approaches using SVI.
result VSGD outperforms Adam and SGD on image classification tasks.
Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.
problem Understanding and controlling the instability of gradient descent in deep learning.
method Introduces the Principal Flow (PF), a continuous time flow that approximates gradient descent dynamics.
result The PF captures divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
problem Adam's fast convergence can lead to local minimums in non-convex problems.
method AdaX exponentially accumulates past gradients to adaptively tune the learning rate.
result AdaX outperforms Adam in various machine learning tasks, including computer vision and natural language processing.
Paper proposes adaptive parameter selection for KGD algorithms.
problem Improving parameter selection for kernel-based gradient descent.
method Integrates bias-variance analysis with splitting method, introduces empirical effective dimension.
result Adaptive parameter selection strategy achieves optimal generalization error bound.
Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
New method improves accuracy in computing implied volatility.
problem Computing implied volatility from the Black-Scholes model.
method Adaptive gradient descent optimizers for numerical computation.
result More accurate results compared to close form approximation and Newton-Raphson method.
Adapting functional gradients improves FGD's practicality and theoretical guarantees.
problem Implementing FGD in practice due to infinite-dimensional functional gradients.
method Adapting the representation of functional gradients.
result Establishes convergence to a stationary point for smooth losses and a global minimizer under smoothness + Polyak-Lojasiewicz condition.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
New method improves neural network robustness to adversarial attacks.
problem Improving adversarial robustness of neural networks.
method Inspired by adaptive control theory, the approach uses persistency of excitation to constrain gradient descent updates.
result Networks trained with the PoE-motivated learning rate schedule are significantly more robust to adversarial attacks.
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
problem Nonconvex machine learning problems with generalized-smoothness.
method Adaptive gradient normalization, independent sampling, and gradient clipping.
result Achieves an O(ε^(-4)) sample complexity for fast convergence.
Improved SGD with AdaGrad stepsizes adapts to unknown parameters and unbounded gradients.
problem Adaptive optimization with unknown parameters and unbounded gradients.
method Stochastic Gradient Descent with AdaGrad stepsizes, without assuming problem parameters or strong global Lipschitz conditions.
result Sharp rates of convergence in both low-noise and high-noise regimes, supporting an affine variance noise model.
Gradient descent with geometrically adapted metrics drives L2 cost to global minimum at uniform rate.
problem Minimizing L2 cost in deep learning networks. method Adapting gradient descent to output layer metric in deep learning.
result Uniform exponential convergence to global minimum in L2 cost. New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
DoWG optimizer automatically adapts to convex and nonsmooth problems without tuning.
problem Optimizing machine learning models efficiently and adaptively.
method DoWG uses a distance-based weighted version of gradient averaging for optimization.
result DoWG achieves convergence rates similar to optimally tuned gradient descent.
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient desce…
We present a strikingly simple proof that two rules are sufficient to automate gradient descent: 1) don't increase the stepsize too fast and 2) don't overstep the local curvature. No need for functional values, no line search, no information about the function except for the gradients. By following these rules, you get…
The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …
Private adaptive methods improve on traditional SGD for convex optimization.
problem Differential privacy constraints in gradient optimization.
method Differentially private variants of SGD and AdaGrad with adaptive stepsizes and non-isotropic clipping.
result Private AdaGrad outperforms private SGD in high-dimensional problems.
Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drasti…
AdaLoss optimizes adaptive learning rates for efficient convergence in various models.
problem Efficiently optimizing adaptive learning rates for gradient descent methods.
method AdaLoss uses loss function information to dynamically adjust step sizes.
result AdaLoss achieves linear convergence in linear regression and robust global convergence in neural networks.
Introduces a new stochastic optimization method for deep learning.
problem Minimizing loss functions in deep neural networks.
method Introduces a second-order stochastic Runge-Kutta method and an adaptive SGD-G2.
result The method yields consistent minimization of loss functions and automatically adjusts learning rates.
We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …
Adaptive sampling improves convergence in heterogeneous distributed optimization.
problem Poor performance of classical SGD and SVRG in heterogeneous distributed settings.
method Adaptive sampling of machines with an adaptive estimate of local Lipschitz constants.
result Significantly accelerates convergence rate from maximum to average Lipschitz constant.
In this paper, we propose an adaptive stopping rule for kernel-based gradient descent (KGD) algorithms. We introduce the empirical effective dimension to quantify the increments of iterations in KGD and derive an implementable early stopping strategy. We analyze the performance of the adaptive stopping rule in the fram…
Gradient descent on normalized networks reveals sparsity preferences.
problem Understanding the inductive bias of gradient descent on normalized neural nets.
method Analysis of gradient descent on weight-normalized smooth homogeneous neural nets, focusing on SWN and EWN.
result EWN causes weights to be updated in a way that prefers asymptotic relative sparsity.
Improved linear regression with privacy and robustness guarantees.
problem Private and robust linear regression with adversarial corruption.
method Differentially private stochastic gradient descent with full-batch gradient descent and adaptive clipping.
result Near optimal sample complexity for both private and robust linear regression.
Characterizes corridors in loss surfaces for gradient-based optimization.
problem Understanding and mitigating training instabilities in gradient-based optimization.
method Characterizes corridors as regions where gradient descent and gradient flow trajectories are linearly related.
result Corridors indicate regions without implicit regularization effects, leading to better learning rate adaptation schemes.
New method reduces communication costs in distributed nonconvex optimization.
problem Large communication costs between central server and local workers in distributed learning.
method Communication-compressed AMSGrad for distributed nonconvex optimization.
result Converges to first-order stationary point with same iteration complexity as vanilla AMSGrad.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
problem The generalization gap between large-batch and small-batch training in deep learning.
method AdAdaGrad introduces adaptive batch size strategies derived from adaptive sampling methods.
result AdAdaGradNorm converges to a first-order stationary point with a rate of O(1/K) in K iterations.
GD with large, adaptive stepsizes achieves optimal risk in logistic regression.
problem Optimizing logistic regression with large stepsizes.
method Gradient Descent with adaptive stepsizes.
result GD achieves minimax optimal convergence in logistic regression.
New method improves scalability of SGD for large datasets.
problem High variance in stochastic gradient descent.
method Adaptive measure reduction with Carathéodory's theorem.
result Improved scalability to high-dimensional spaces.
New adaptive importance samplers improve stability and accuracy.
problem Improving the stability and accuracy of importance sampling estimators.
method Introducing AdaOAIS, a new adaptive importance sampler using adaptive optimisers to address the instability of OAIS.
result AdaOAIS leads to stable importance sampling estimators in practice.
New SGD method uses adaptive sampling to converge faster in non-convex problems.
problem Non-convex optimization problems with noisy gradients.
method Adaptive coordinate sampling in stochastic gradient descent (SGD).
result Almost sure convergence and non-asymptotic bounds established.
Adapts SGD to noise and problem specifics for faster convergence.
problem Minimizing smooth, strongly-convex functions with varying noise and problem constants.
method Adaptive SGD with exponentially decreasing step-sizes, Nesterov acceleration, and stochastic line-search.
result Achieves near-optimal convergence rates without knowing noise or problem specifics.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
problem Optimizing non-convex functions on Riemannian manifolds.
method Batch size adaptation in R-SVRG, R-SRG, and R-SPIDER.
result Achieves lower total complexities for various non-convex functions.