arXiv research

Adaptive momentum method solves non-convex min-max problems.

Paper analyzes high probability convergence of adaptive SGD with momentum.

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and…

This paper presents generalized momentum mappings for covariant Hamiltonian field theories. The new momentum mappings arise from a generalization of symplectic geometry to $L_VY$, the bundle of vertically adapted linear frames over the bundle of field configurations $Y$. Specifically, the generalized field momentum obs…

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…

The article examines in some detail the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size and slow adaptation regime. The results establish that momentum methods are equivalent to the standard stochastic gradient method with a re-scaled (larger) step-si…

Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…

Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter $m$ which is always suggested to be set to less than $1$. Although the choice…

Improved loss scaling for stochastic momentum algorithms in high dimensions.

Efficient momentum-based methods for reinforcement learning with improved sample complexity.

New algorithm improves stability of optimization algorithms by adapting step-size.

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive $q$-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …

An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive stepsize of each individual parameter. The dynamic relaxation is adapted for stochast…

Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key co…

Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD …

This work provides formal guarantees for heuristic optimization methods in machine learning.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …

Paper proves suboptimal convergence rate of last iterate for SGDM.

GIST adapts HMC by tuning parameters based on position and momentum.

The paper proves positive energy-momentum theorems for charged AdS initial data sets.

Intelligent Momentum Transformer outperforms traditional trading strategies.

AB dynamically scales gradients to mitigate asynchronous training delays.

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…

CAdam optimizes online learning by adapting to distribution shifts and noise.

Enhanced Momentum Transformer outperforms traditional trading strategies.

The abstract explores a new wave equation linking quantum mechanics and complex adaptive systems.

New insights into SGD and SGD-M in high dimensions.

New SGD algorithm finds critical points faster with second-order corrections.

This work is a part of ICLR Reproducibility Challenge 2019, we try to reproduce the results in the conference submission PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks. Adaptive gradient methods proposed in past demonstrate a degraded generalization performance than …

We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives. Then, we consider the case of objectives with bounded second derivative and show that in this case a small tweak to the momentum formula allows normalized SGD with mom…

Mime algorithm improves federated learning by adapting centralized methods.

MiM-StocR combines momentum indicators and adaptive ranking loss for better stock recommendation.

The study examines the dynamic behavior of RMSprop and Adam algorithms.

A new method uses trivialized momentum to generate data on Lie groups.

Introduces VSMD to improve generative diffusion processes without high costs.

A fast method for training linear classifiers maximizes margins.

FedMCC learns from distributed data to cluster and extract features.

Rod flow models Adam's behavior at the edge of stability.

Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…

Derives effective continuous dynamics for adaptive SGD methods.

A new optimizer preserves orthogonality constraints on matrices efficiently.

NoxTrader predicts stock returns using LSTM for profitable trading.

Integrates momentum into SMoE for improved stability and robustness.

This paper investigates the time-varying risk-premium relation of the Chinese stock markets within the framework of cross-sectional momentum and contrarian effects by adopting the Capital Asset Pricing Model and the French-Fama three factor model. The evolving arbitrage opportunities are also studied by quantifying the…

To overcome the oscillation problem in the classical momentum-based optimizer, recent work associates it with the proportional-integral (PI) controller, and artificially adds D term producing a PID controller. It suppresses oscillation with the sacrifice of introducing extra hyper-parameter. In this paper, we start by …

The adaptive momentum method (AdaMM), which uses past gradients to update descent directions and learning rates simultaneously, has become one of the most popular first-order optimization methods for solving machine learning problems. However, AdaMM is not suited for solving black-box optimization problems, where expli…