arXiv research

This paper analyzes and improves monotonic accelerated algorithms like M-NAG and M-FISTA.

Paper improves learning rates for SGD and NAG.

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…

FedNAG improves federated learning accuracy and reduces training time.

Improved optimization guarantees for deep learning models with Nesterov acceleration.

GRU models with Adam optimizer outperform other combinations in stock market forecasting.

The article considers smooth optimization of functions on Lie groups. By generalizing NAG variational principle in vector space (Wibisono et al., 2016) to Lie groups, continuous Lie-NAG dynamics which are guaranteed to converge to local optimum are obtained. They correspond to momentum versions of gradient flow on Lie …

SRSGD improves DNN training speed and accuracy.

New insights show NAG and FISTA converge linearly without knowing strong convexity modulus.

Proposes momentum methods for Lie groups, improving on classical algorithms.

New method solves convex optimization faster than NAG.

This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speak…

This research accelerates sampling methods using Nesterov's Acceleration.

A new family of momentum coefficients improves the convergence rate of accelerated algorithms.

NeCPD improves online tensor decomposition using SGD with Hessian analysis and NAG.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or …

GD and NAG accelerate matrix factorization and neural networks.

The overall performance or expected excess risk of an iterative machine learning algorithm can be decomposed into training error and generalization error. While the former is controlled by its convergence analysis, the latter can be tightly handled by algorithmic stability. The machine learning community has a rich his…

Paper studies stochastic optimization methods with momentum, proving convergence and avoiding traps.

Unified framework for accelerated Perceptron and related problems.

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…

While Standard gradient descent is one very popular optimisation method, its convergence cannot be proven beyond the class of functions whose gradient is globally Lipschitz continuous. As such, it is not actually applicable to realistic applications such as Deep Neural Networks. In this paper, we prove that its backtra…

The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successe…

Abstract commensurators linked to topological models of solenoids.

Improved backpropagation with consequentialism weight updates for neural networks.

Improved neural-ODE for faster convergence and stability.

GOLS-I automatically determines learning rates for various neural network training algorithms.

Optimization algorithms help overparameterized neural networks achieve high performance.

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

We study the Hilbert manifold structure on $T_{0}(1)$ -- the connected component of the identity of the Hilbert manifold T(1). We characterize points on $T_{0}(1)$ in terms of Bers and pre-Bers embeddings, and prove that the Grunsky operators $B_{1}$ and $B_{4}$, associated with the points in $T_{0}(1)$ via conformal w…

Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the rol…

Paper shows linear convergence of ISTA and FISTA for ill-conditioned images.