The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “momentum parameters”
The paper examines geometric curvatures in generalized Riemannian spaces.
problem Understanding the physical meaning of scalar curvatures in generalized Riemannian spaces.
method Developed Madsen's formulae for pressures and energy-densities, analyzed with different concepts of generalized Riemannian spaces.
result Linearities of energy-momentum tensor, pressure, energy-density, and state-parameter are examined.
The conformal method is a technique for finding Cauchy data in general relativity solving the Einstein constraint equations, and its parameters include a conformal class, a conformal momentum (as measured by a densitized lapse), and a mean curvature. Although the conformal method is successful in generating constant me…
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…
Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter which is always suggested to be set to less than . Although the choice…
We introduce the concept of spontaneous symmetry breaking to arbitrage modeling. In the model, the arbitrage strategy is considered as being in the symmetry breaking phase and the phase transition between arbitrage mode and no-arbitrage mode is triggered by a control parameter. We estimate the control parameter for mom…
New loss function connects learning rate and momentum.
problem Finding optimal learning rate and momentum empirically.
method Proposes a new information-theoretical loss function.
result Loss, learning rate, and momentum are closely connected.
Analysis of momentum methods on quadratic models, showing SGD's superiority.
problem Analysis of stochastic gradient algorithms with momentum on quadratic models.
method Inspired by random matrix theory, exact characterization of loss values.
result Stochastic heavy-ball momentum does not improve over SGD in the strongly convex setting.
A new principle minimizes residual and introduces momentum to improve PDE solution dynamics.
problem Ill-conditioning in Dirac-Frenkel residual minimization leads to non-unique parameter dynamics.
method Introduces a history variable (momentum) to select better-conditioned parameter velocities, preserving residual minimization while promoting smooth parameter evolutions.
result The approach leads to increased robustness in singular and near-singular PDE solution regimes.
The purpose of this paper is to indicate that the recently proposed Momentum fractional least mean squares (mFLMS) algorithm has some serious flaws in its design and analysis. Our apprehensions are based on the evidence we found in the derivation and analysis in the paper titled: \textquotedblleft \textit{Momentum frac…
In this paper we study the problem of convergence and generalization error bound of stochastic momentum for deep learning from the perspective of regularization. To do so, we first interpret momentum as solving an -regularized minimization problem to learn the offsets between arbitrary two successive model para…
Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithmquant-ph
New algorithm Momentum-QNG improves optimization of quantum circuits.
problem Optimizing variational quantum circuits to avoid local minima.
method Applied Langevin dynamics to QNG, introducing momentum term.
result Momentum-QNG outperforms basic QNG and other optimizers.
Unified algorithm for stochastic optimization with time-varying momentum converges under general conditions.
problem Optimizing functions with time-varying gradients and biases.
method Unified algorithm using a time-varying momentum term.
result Convergence of the unified algorithm under general conditions.
A new accelerated method with simpler momentum update rules.
problem Optimizing parameters in machine learning models.
method Proposes a novel accelerated stochastic gradient method with simpler momentum update rules.
result The method outperforms Sgdm and Adam in practical problems.
This study uses continuous-time analysis to understand how momentum affects the optimisation of diagonal linear networks.
problem The effect of momentum on the optimisation trajectory of gradient descent.
method Leveraging a continuous-time approach to analyze momentum gradient descent with step size γ and momentum parameter β.
result Small values of λ help recover sparse solutions in overparametrised regression settings.
The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.
problem Understanding and mitigating the dying ReLU problem in neural networks.
method Empirical analysis and theoretical modeling of a discrete-time linear autonomous system.
result Target variance plays a crucial role in the dying ReLU phenomenon, and momentum exacerbates this issue.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
Momentum SGD fails to track nonstationary optima due to drift amplification.
problem Tracking nonstationary optima in stochastic optimization.
method Theoretical analysis of SGD and momentum variants under strong convexity and smoothness.
result Momentum incurs a drift-amplification penalty that diverges as the momentum parameter approaches 1, leading to systematic lag.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
Improved convergence for Polyak steps with momentum in smooth convex optimization.
problem Optimizing smooth strongly convex functions with limited information.
method Polyak steps with momentum, derived for accelerated gradient method.
result Convergence guarantees for accelerated gradient method with Polyak steps and momentum.
Laprop separates Adam's momentum and adaptivity to improve stability and speed.
problem Unnecessary coupling between Adam's momentum and adaptivity leads to instability and divergence.
method Proposes Laprop, a method that decouples momentum and adaptivity.
result Laprop consistently improves speed and stability over Adam on various tasks.
An algorithm is presented for momentum gradient descent optimization based on the first-order differential equation of the Newtonian dynamics. The fictitious mass is introduced to the dynamics of momentum for regularizing the adaptive stepsize of each individual parameter. The dynamic relaxation is adapted for stochast…
New insights into using momentum for non-convex optimization.
problem Improving training of non-convex models like deep neural networks.
method Developed a Lyapunov analysis of SGD with momentum using stochastic primal averaging.
result Precise conditions under which SGD+M outperforms SGD and optimal hyper-parameter schedules.
The paper defines cross-section continuity for angular momentum definitions and finds the CWY definition valid.
problem Defining angular momentum at null infinity and ensuring its continuity across different cross-sections.
method Introducing cross-section continuity as a criterion and proving it for specific angular momentum definitions.
result The Chen-Wang-Yau definition of angular momentum satisfies cross-section continuity, while the Compere-Nichols modification does not.
Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the rol…
Super-acceleration of gradient descent with momentum improves loss function minimization.
problem Minimizing loss functions in machine learning.
method Extending Nesterov acceleration by using gradients at multiple steps ahead.
result Super-acceleration of the momentum algorithm is beneficial for various loss landscapes and tasks.
The study analyzes momentum-based optimization algorithms from dynamical systems perspective.
problem Understanding convergence rates of momentum-based optimization algorithms.
method Exploits dynamical systems, control theory, and symplectic perspectives to analyze convergence rates.
result Provides closed-form expressions relating algorithm parameters to convergence rates.
Proposes EDM algorithm to accelerate model training in distributed networks.
problem Hindered effectiveness of distributed stochastic optimization algorithms due to data heterogeneity and network sparsity.
method Introduces Exact-Diffusion with Momentum (EDM) algorithm, incorporating momentum techniques to mitigate bias and enhance convergence rate.
result EDM algorithm converges sub-linearly to the optimal solution, radius independent of data heterogeneity, for non-convex objective functions.
In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these metho…
Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to ac…
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient . Large values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typic…
A new correction term improves sample efficiency in deep reinforcement learning.
problem Momentum accumulation in TD learning leads to doubly stale gradients.
method Proposed a correction term to address the issue of doubly stale gradients.
result Improves sample efficiency in policy evaluation.
It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reache…
STORM-PG uses momentum for faster policy gradient updates.
problem Improving policy gradient methods for reinforcement learning.
method Introduces STORM-PG, a SARAH-based algorithm with exponential moving average.
result Achieves sample complexity, matching best-known rate.
Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noisemath.OC
New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.
problem Optimization problems with heavy-tailed noise and weakly average smoothness.
method Normalized stochastic first-order methods with Polyak, multi-extrapolated, and recursive momentum.
result First-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise.
The article examines in some detail the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size and slow adaptation regime. The results establish that momentum methods are equivalent to the standard stochastic gradient method with a re-scaled (larger) step-si…
New insights into training machine learning models with momentum.
problem Lack of theoretical understanding on the generalization error of momentum-based methods.
method Analyzed modified momentum-based update rule (SGDEM) for smooth Lipschitz loss functions.
result SGDEM admits an upper-bound on the generalization error for smooth Lipschitz loss functions.
Constructs initial data for multiple black holes with specified ADM parameters.
problem Forming multiple black holes with specific ADM parameters.
method Smooth, asymptotically flat vacuum initial data with prescribed ADM energy, momentum, and angular momentum.
result Maximal development of data results in spacetimes containing multiple black holes.
Asynchronous methods are widely used in deep learning, but have limited theoretical justification when applied to non-convex problems. We show that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding a momentum-like term to the SGD iteration. Our result does not assume convexity …
We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods. By simply modifying plain IHT, we investigate its convergence behavior on convex optimization criteria with non-convex constraints, under standard assumptions. In diverse scenaria, we observe that …
MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
New factorial power constants improve optimization convergence rates.
problem Optimization convergence rates depend on various constants.
method Proposes using factorial powers for defining these constants.
result Factorial powers simplify or improve convergence rates of optimization methods.
Integrates momentum into SMoE for improved stability and robustness.
problem Stable training and robustness to data contamination in SMoE models.
method Established a connection between SMoE dynamics and gradient descent, then integrated momentum.
result MomentumSMoE is more stable and robust than SMoE.
The paper studies momentum-based minimization for Ginzburg-Landau on Euclidean spaces and graphs.
problem Minimizing the Ginzburg-Landau functional on Euclidean spaces and graphs.
method Momentum-based minimization using a convex-concave splitting-based FISTA-type time discretization.
result Momentum can lead to faster convergence if the time step size is large but not too large.
AdamS uses momentum as a denominator to optimize LLMs efficiently.
problem Optimizing large language models (LLMs) with efficient and effective methods.
method AdamS introduces a novel denominator based on the root of the weighted sum of squares of momentum and current gradient.
result AdamS achieves superior optimization performance with minimal memory and compute requirements.
Fine-grained analysis of gradient descent with momentum provides modified loss equations.
problem Understanding the dynamics of gradient descent with momentum.
method Fine-grained analysis and derivation of modified loss equations.
result Global approximation bounds and continuous modified equations for HB.
Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficult…