Momentum is a popular technique to accelerate the convergence in practical training, and its impact on convergence guarantee has been well-studied for first-order algorithms. However, such a successful acceleration technique has not yet been proposed for second-order algorithms in nonconvex optimization.In this paper, …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “momentum schemes”
Paper develops momentum schemes with variance reduction for non-convex composition optimization.
problem Lack of convergence guarantee and efficient momentum design in existing algorithms.
method Develops various momentum schemes with SPIDER-based variance reduction.
result Achieves near-optimal sample complexity and linear convergence rate.
A new Bayesian modeling method is proposed by combining the maximization of the marginal likelihood with a momentum-space renormalization group transformation for Gaussian graphical models. Moreover, we present a scheme for computint the statistical averages of hyperparameters and mean square errors in our proposed met…
The study analyzes momentum-based optimization algorithms from dynamical systems perspective.
problem Understanding convergence rates of momentum-based optimization algorithms.
method Exploits dynamical systems, control theory, and symplectic perspectives to analyze convergence rates.
result Provides closed-form expressions relating algorithm parameters to convergence rates.
This paper analyzes momentum Q-learning with finite-sample guarantees.
problem Improving Q-learning performance with momentum schemes.
method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.
result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyper…
Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, espec…
This paper analyzes two Lie group momentum optimization algorithms and their convergence rates.
problem Optimizing functions on Lie groups using momentum-based dynamics.
method Investigates Lie Heavy-Ball and Lie NAG-SC algorithms, quantifying their convergence rates under smoothness and convexity assumptions.
result Lie NAG-SC accelerates optimization over the momentumless case, while Lie Heavy-Ball does not.
New HMC method uses asymmetrical momentum distributions and improves performance.
problem Rigorous convergence guarantees for HMC with Gaussian momentum distributions.
method New convergence analysis for HMC with general asymmetrical momentum distributions, proposing AD-HMC.
result AD-HMC exhibits geometric convergence in Wasserstein distance under certain conditions.
SRSGD improves DNN training speed and accuracy.
problem Training deep neural networks is computationally expensive and slow.
method Scheduled Restart SGD (SRSGD) combines NAG momentum with momentum reset.
result SRSGD achieves better error rates with fewer training epochs.
FedMCC learns from distributed data to cluster and extract features.
problem Learning from distributed data and clustering.
method Federated Momentum Contrastive Clustering (FedMCC) framework.
result FedMCC outperforms existing methods in linear evaluation and semi-supervised learning.
New methods reduce constraint violations to certainty in stochastic optimization.
problem Finding a point with certain constraint satisfaction and near-stationarity.
method Single-loop variance-reduced stochastic first-order methods with truncated momentum schemes.
result Achieves strong convergence guarantees for -stochastic stationary points with certain constraint satisfaction.
MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization. However, SPIDER uses an accuracy-dependent stepsize that slows down the convergence in practice, and cannot handle objec…
Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. …
Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …
The study finds that factor momentum is significant only at short lags compared to stock momentum.
problem Investigating the relationship between factor momentum and stock momentum.
method Replicated earlier findings and conducted a spanning test controlling for stock momentum and factor exposure.
result Factor momentum is significant only at short lags after controlling for stock momentum and factor exposure.
In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for convex optimization, but it does not work well for nonconvex optimization typic…
We test the price momentum effect in the Korean stock markets under the momentum universe shrinkage to subuniverses of the KOSPI 200. Performance of the momentum strategy is not homogeneous with respect to change of the momentum universe. It is found that some submarkets generate the higher momentum returns than other …
Introduces homotopy momentum sections on multisymplectic manifolds.
problem No specific problem stated; focuses on introducing a new concept.
method Introduces a new concept of homotopy momentum sections on multisymplectic manifolds.
result Shows that a gauged nonlinear sigma model with Wess-Zumino term has homotopy momentum section structure.
Customer Momentumq-fin.PR
Customer momentum is a positive relationship between a firm's returns and past returns of its customers.
problem Understanding the relationship between a firm's returns and its customers' past returns.
method Examined customer momentum using a long-short equally-weighted decile portfolio and Fama-French factor models.
result Customer momentum generates significant monthly returns and is statistically significant.
This paper examines momentum spillover across multiple asset classes using only pricing data.
problem Challenges in studying momentum spillover across diverse asset classes due to lack of common characteristics.
method Utilised a linear and interpretable graph learning model to reveal momentum spillover network.
result Network momentum strategy yields a Sharpe ratio of 1.5 and an annual return of 22%.
The paper analyzes how hyperparameters affect SGD with momentum's convergence rate.
problem The role of hyperparameters in SGD with momentum's convergence rate.
method Theoretical analysis using a hyperparameters-dependent stochastic differential equation (hp-dependent SDE).
result The optimal linear rate of convergence depends on both the learning rate and the momentum coefficient.
We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced op…
This paper presents generalized momentum mappings for covariant Hamiltonian field theories. The new momentum mappings arise from a generalization of symplectic geometry to , the bundle of vertically adapted linear frames over the bundle of field configurations . Specifically, the generalized field momentum obs…
The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball an…
We give a detailed discussion about existence and uniqueness of Lu's momentum map. More precisely, we introduce the infinitesimal momentum map, and we study its properties. This allows us to describe the theory of reconstruction of the momentum map from the infinitesimal one. We provide the conditions for the uniquenes…
Momentum ResNets improve ResNets' memory efficiency.
problem Memory inefficiency in deep residual neural networks (ResNets).
method Adding a momentum term to the forward rule of ResNets to make them invertible.
result Momentum ResNets can learn any linear mapping up to a multiplicative factor, improving memory efficiency.
The paper analyzes how momentum affects convergence in stochastic gradient methods.
problem Lack of clear understanding of momentum's impact on convergence and performance.
method Unified analysis of several popular algorithms using the QHM formulation.
result Provides practical guidelines for setting learning rate and momentum parameters.
We introduce various quantitative and mathematical definitions for price momentum of financial instruments. The price momentum is quantified with velocity and mass concepts originated from the momentum in physics. By using the physical momentum of price as a selection criterion, the weekly contrarian strategies are imp…
This paper studies accelerations in Q-learning algorithms. We propose an accelerated target update scheme by incorporating the historical iterates of Q functions. The idea is conceptually inspired by the momentum-based accelerated methods in the optimization theory. Conditions under which the proposed accelerated algor…
Adapting momentum from optimization to reinforcement learning.
problem Improving the convergence and stability of reinforcement learning algorithms.
method Introducing Momentum Value Iteration (MoVI) by incorporating an average of consecutive state-action value functions, inspired by the concept of momentum in optimization.
result MoVI improves the convergence and stability of reinforcement learning algorithms, as demonstrated by experiments on Atari games.
Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithmquant-ph
New algorithm Momentum-QNG improves optimization of quantum circuits.
problem Optimizing variational quantum circuits to avoid local minima.
method Applied Langevin dynamics to QNG, introducing momentum term.
result Momentum-QNG outperforms basic QNG and other optimizers.
We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to…
Momentum speeds up evolutionary processes in machine learning.
problem Accelerating convergence in evolutionary dynamics.
method Combining momentum from machine learning with evolutionary dynamics using information divergences as Lyapunov functions.
result Momentum accelerates convergence of evolutionary dynamics, including the replicator equation and Euclidean gradient descent.
Optimizes web page freshness with limited crawling frequencies.
problem Maximize local cache freshness given crawling frequency constraints.
method Three novel online estimation schemes for page change rates.
result Convergent algorithms for estimating page change rates.
Contact manifolds' momentum polytopes are convex.
problem Understanding the structure of contact manifolds.
method Using isomorphism to toric varieties.
result Momentum polytopes of contact manifolds are convex.
Unified model learns from both time-series and cross-sectional momentum features.
problem Separate time-series and cross-sectional momentum strategies do not consider concurrent relationships.
method Spatio-Temporal Momentum strategies using neural networks to combine both types of momentum.
result Simple neural network with single fully connected layer generates trading signals for all assets.
SMG combines shuffling and momentum for non-convex optimization.
problem Non-convex finite-sum optimization problems.
method Shuffling Gradient-based method with momentum.
result Established state-of-the-art convergence rates for SMG.
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…
One has not any conventional energy-momentum conservation law in Lagrangian field theory, but relations involving different stress-energy-momentum tensors associated with different connections. It is not obvious how to choose the true energy-momentum tensor. This problem is solved in the framework of the multimomentum …
Constructibility of momentum maps and linear variation for singular symplectic reduced spacesmath.SG
The paper extends a theorem about momentum maps to singular symplectic spaces.
problem Extending a theorem about momentum maps to singular symplectic spaces.
method Using integral affine stratification and equivariant locally trivial fibrations, the paper extends the linear variation theorem to singular values of the momentum map.
result Cohomology classes of symplectic forms on reduced spaces vary linearly within strata.
Momentum improves deep learning generalization by stabilizing noise and learning features.
problem Improving generalization in deep learning models.
method Empirical and theoretical analysis of gradient descent with momentum (GD+M) vs. gradient descent (GD) in binary classification tasks.
result GD+M outperforms GD in generalization, especially in datasets with shared features and varying margins.
Generalizes momentum map to Courant algebroid for constrained mechanics.
problem Generalizing momentum map to new geometric structures.
method Generalized momentum section on Lie algebroid to Courant algebroid, constructed cohomological formulations.
result Identified momentum section in constrained Hamiltonian mechanics with Courant algebroid symmetry.