Laprop separates Adam's momentum and adaptivity to improve stability and speed.
problem Unnecessary coupling between Adam's momentum and adaptivity leads to instability and divergence.
method Proposes Laprop, a method that decouples momentum and adaptivity.
result Laprop consistently improves speed and stability over Adam on various tasks.
Adaptive momentum method solves non-convex min-max problems.
problem Non-convex min-max optimization problems in training generative adversarial networks.
method Proposes an adaptive momentum algorithm for non-convex min-max optimization.
result Establishes non-asymptotic convergence rates for the proposed algorithm.
Paper analyzes high probability convergence of adaptive SGD with momentum.
problem Theoretical understanding of adaptive SGD with momentum in nonconvex settings is incomplete.
method High probability analysis under weak assumptions.
result First high probability convergence proof for gradients to zero in Delayed AdaGrad with momentum.
This paper explains why Adam generalizes worse than SGD by analyzing its components.
problem Understanding why Adam generalizes worse than Stochastic Gradient Descent (SGD).
method Diffusion theoretical framework to disentangle the effects of Adaptive Learning Rate and Momentum.
result Adaptive Learning Rate helps escape saddle points but not select flat minima, while Momentum provides a drift effect to help pass through saddle points.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
This paper presents generalized momentum mappings for covariant Hamiltonian field theories. The new momentum mappings arise from a generalization of symplectic geometry to LVY, the bundle of vertically adapted linear frames over the bundle of field configurations Y. Specifically, the generalized field momentum obs…
Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…
The article examines in some detail the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size and slow adaptation regime. The results establish that momentum methods are equivalent to the standard stochastic gradient method with a re-scaled (larger) step-si…
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…
Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter m which is always suggested to be set to less than 1. Although the choice…
Improved loss scaling for stochastic momentum algorithms in high dimensions.
problem Improving loss scaling for stochastic momentum algorithms in high dimensions.
method Dimension-adapted Nesterov acceleration (DANA) scales momentum hyperparameters based on model size and data complexity.
result DANA improves loss scaling exponents across various data and target complexities.
Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.
problem Reducing batch size requirements for SGD in non-convex optimization.
method Normalized SGD with momentum, adaptive method for small gradient variance.
result Normalized SGD with momentum achieves ε-critical points in O(1/ε3.5) iterations. Efficient momentum-based methods for reinforcement learning with improved sample complexity.
problem Efficient reinforcement learning algorithms for non-concave performance functions.
method Adaptive momentum-based policy gradient methods using variance reduction and importance sampling techniques.
result Both IS-MBPG and HA-MBPG reach the best known sample complexity of O(ε−3) for finding an ε-stationary point. New algorithm improves stability of optimization algorithms by adapting step-size.
problem Optimization algorithms' effectiveness is sensitive to step-size hyperparameters.
method Adapts NGN step-size method with momentum to enhance stability.
result Achieves convergence rate of O(1/√K) without restrictive assumptions.
We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive q-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …
An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive stepsize of each individual parameter. The dynamic relaxation is adapted for stochast…
Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key co…
Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD …
This work provides formal guarantees for heuristic optimization methods in machine learning.
problem Lack of theoretical understanding of heuristic optimization methods in machine learning.
method Analysis and formal guarantees for AdaGrad, SGD with exponential and cosine step sizes, and momentum methods.
result First formal guarantees for AdaGrad and SGD variants, including convergence and adaptivity to noise.
We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …
Paper proves suboptimal convergence rate of last iterate for SGDM.
problem Proves suboptimal convergence rate of last iterate for SGDM.
method Focuses on convergence rate of last iterate of SGDM, introduces Follow-The-Regularized-Leader-based algorithms.
result Shows optimal convergence rate of last iterate for unconstrained convex stochastic optimization problems.
GIST adapts HMC by tuning parameters based on position and momentum.
problem Locally adaptive sampling in Hamiltonian Monte Carlo.
method GIST uses Gibbs sampling to adaptively tune HMC parameters.
result GIST improves sampling efficiency for high-dimensional models.
The paper proves positive energy-momentum theorems for charged AdS initial data sets.
problem Proving positive energy-momentum theorems for charged asymptotically AdS initial data sets.
method Introducing a charged energy-momentum functional and establishing positive theorems under a dominant energy condition.
result The charged energy-momentum functional is non-negative on a natural real cone.
Intelligent Momentum Transformer outperforms traditional trading strategies.
problem Improving time-series momentum and mean-reversion trading strategies.
method Attention-based deep-learning architecture (Momentum Transformer) combining attention and LSTM.
result Momentum Transformer outperforms benchmarks and adapts to new market regimes.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…
CAdam optimizes online learning by adapting to distribution shifts and noise.
problem Challenges in online learning data, including distribution shifts and noise, affect Adam's performance.
method CAdam uses a confidence-based approach to assess the consistency between momentum and gradients before updating parameters.
result CAdam outperforms other optimizers in various settings with distribution shift or noise.
Enhanced Momentum Transformer outperforms traditional trading strategies.
problem Improving trading performance in equities with evolving market conditions.
method Building a Momentum Transformer using an attention mechanism combined with LSTM, capturing long-term dependencies and transaction costs.
result Average returns of 4.14% and Sharpe ratio of 1.12, similar to original results but with higher volatility.
The abstract explores a new wave equation linking quantum mechanics and complex adaptive systems.
problem Understanding the underlying mechanism of distribution formation in complex quantum entanglement.
method Exploring the logical relationship between Schrödinger's wave equation and Shi's trading volume-price wave equation in finance.
result A non-localized wave equation in quantum mechanics reveals the invariance of interaction as a universal law.
New insights into SGD and SGD-M in high dimensions.
problem Understanding and comparing SGD and SGD-M in high-dimensional settings.
method Developed high-dimensional scaling limits for SGD-M and online SGD, examining their dynamics and performance.
result SGD-M amplifies high-dimensional effects, potentially degrading performance compared to online SGD.
New SGD algorithm finds critical points faster with second-order corrections.
problem Finding critical points in non-convex optimization efficiently.
method Uses Hessian-vector products to correct momentum bias in SGD.
result Finds ε-critical points in O(ε−3) time. This work is a part of ICLR Reproducibility Challenge 2019, we try to reproduce the results in the conference submission PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks. Adaptive gradient methods proposed in past demonstrate a degraded generalization performance than …
Mime algorithm improves federated learning by adapting centralized methods.
problem Mitigating client drift in federated learning.
method Combines control variates and server-level statistics to adapt centralized algorithms to federated learning.
result Mime outperforms any centralized method in federated learning.
MiM-StocR combines momentum indicators and adaptive ranking loss for better stock recommendation.
problem Lack of simultaneous short-term trend and ranking prediction in stock recommendation models.
method Integrates momentum indicators and proposes Adaptive-k ApproxNDCG for ranking optimization.
result MiM-StocR outperforms state-of-the-art MTL baselines in stock recommendation.
The study examines the dynamic behavior of RMSprop and Adam algorithms.
problem Understanding the training loss curve patterns of adaptive gradient algorithms.
method Careful numerical experiments and theoretical explanations using the signGD flow.
result Adam converges smoother and faster when momentum factors are close to each other.
Super-acceleration of gradient descent with momentum improves loss function minimization.
problem Minimizing loss functions in machine learning.
method Extending Nesterov acceleration by using gradients at multiple steps ahead.
result Super-acceleration of the momentum algorithm is beneficial for various loss landscapes and tasks.
A new method uses trivialized momentum to generate data on Lie groups.
problem Generating data on Lie groups with high fidelity and efficiency.
method Introducing an auxiliary momentum variable that stays in a fixed vector space, and using a manifold preserving integrator.
result Achieves state-of-the-art performance on protein and RNA torsion angle generation and high-dimensional Lie groups.
Introduces VSMD to improve generative diffusion processes without high costs.
problem High training costs and scalability issues in generative diffusion processes.
method Introduces variational Schrödinger momentum diffusion (VSMD) with adaptively transport-optimized variational scores and critical-damping transform.
result Efficiently generates anisotropic shapes while maintaining transport efficacy, outperforming alternatives.
A fast method for training linear classifiers maximizes margins.
problem Training linear classifiers with maximum margins.
method Momentum-based gradient method derived from convex dual with Nesterov acceleration.
result Exponentially faster convergence rate compared to standard methods.
FedMCC learns from distributed data to cluster and extract features.
problem Learning from distributed data and clustering.
method Federated Momentum Contrastive Clustering (FedMCC) framework.
result FedMCC outperforms existing methods in linear evaluation and semi-supervised learning.
Rod flow models Adam's behavior at the edge of stability.
problem Modeling adaptive gradient methods like Adam at the edge of stability.
method Extended rod flow to Adam, considering parameters, first moment, and second moment as variables.
result Rod flow accurately tracks Adam's behavior through the edge-of-stability regime.
Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
A new optimizer preserves orthogonality constraints on matrices efficiently.
problem Optimization on Stiefel manifold with orthogonality constraints.
method Interplay between continuous and discrete dynamics leading to a gradient-based optimizer with momentum.
result The method optimizes matrices on Stiefel manifold efficiently and accurately.
NoxTrader predicts stock returns using LSTM for profitable trading.
problem Predicting profitable stock returns for quantitative trading.
method LSTM model for time-series analysis of stock data.
result Improved investment return from -60% to 325%.
Integrates momentum into SMoE for improved stability and robustness.
problem Stable training and robustness to data contamination in SMoE models.
method Established a connection between SMoE dynamics and gradient descent, then integrated momentum.
result MomentumSMoE is more stable and robust than SMoE.
This paper investigates the time-varying risk-premium relation of the Chinese stock markets within the framework of cross-sectional momentum and contrarian effects by adopting the Capital Asset Pricing Model and the French-Fama three factor model. The evolving arbitrage opportunities are also studied by quantifying the…
To overcome the oscillation problem in the classical momentum-based optimizer, recent work associates it with the proportional-integral (PI) controller, and artificially adds D term producing a PID controller. It suppresses oscillation with the sacrifice of introducing extra hyper-parameter. In this paper, we start by …