arXiv research

A new accelerated method with simpler momentum update rules.

We found that factors decay over time, with momentum fitting best.

Demon improves neural network training with a decaying momentum approach.

Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning

Classifies scalar-flat toric Kähler instantons in 4D.

New learning rate schedule improves deep learning performance.

Stochastic momentum methods trade compute efficiency for serial runtime.

New memory effect discovered in gravitational wave behavior.

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reache…

AdamNX improves Adam's stability by adjusting its learning rate.

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to ac…

The paper analyzes dynamics of momentum in high dimensions with sparse updates.

Lion optimizer performs well in training AI models with memory efficiency.

Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.

Study on massless Vlasov equation on Reissner-Nordström spacetimes, showing decay rates and non-decay phenomena.

AdamP optimizes momentum-based optimizers for scale-invariant weights, improving model performance.

New proof removes decay assumptions for spacetime positive mass theorem.

Fine-grained analysis of gradient descent with momentum provides modified loss equations.

In 1996, Huisken-Yau proved that every three-dimensional Riemannian manifold can be uniquely foliated near infinity by stable closed surfaces of constant mean curvature (CMC) if it is asymptotically equal to the (spatial) Schwarzschild solution. Later, their decay assumptions were weakened by Metzger, Huang, Eichmair-M…

The momentum ray transform $I^k$ integrates a rank $m$ symmetric tensor field $f$ over lines of ${\R}^n$ with the weight $t^k$: $ (I^k\!f)(x,ξ)=\int_{-\infty}^\infty t^kłf(x+tξ),ξ^m\r\,dt. $ We give the range characterization for the operator $f\mapsto(I^0\!f,I^1\!f,\dots, I^m\!f)$ on the Schwartz space of rank $m$ smo…

Future stability of FLRW solutions in expanding 3D space is shown for compact perturbations.

Polyak's momentum accelerates training of neural networks.

A new method to improve deep neural networks using weight rescaling.

Study reveals dynamics of neural networks with normalization, weight decay, and SGD.

New analysis shows SNG's effectiveness in small samples.

We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…

We study coordinate-invariance of some asymptotic invariants such as the ADM mass or the Chruściel-Herzlich momentum, given by an integral over a "boundary at infinity". When changing the coordinates at infinity, some terms in the change of integrand do not decay fast enough to have a vanishing integral at infinity; bu…

New method SF-AdamW trains large models without decay phases or memory overhead.

We show that the Yang-Mills quantum field theory with momentum and spacetime cutoffs in four Euclidean dimensions is equivalent, term by term in an appropriately resummed perturbation theory, to a Fermionic theory with nonlocal interaction terms. When a further momentum cutoff is imposed, this Fermionic theory has a co…

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …

A technique identifies memoryless algorithms approximating memory-dependent optimization methods.

New magnetic memory effects found in gravitational waves and memory.

SGDm with fixed step-size diverges under covariate shift, similar to a parametric oscillator.

Closed and broken electromagnetic orbits in Kerr-Newman spacetime

A new optimizer d-AmsGrad improves deep learning for robot learning in non-stationary problems.

Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, w…

Paper solves Einstein vacuum equations gluing problem with applications.

The study finds that factor momentum is significant only at short lags compared to stock momentum.

Recent literature on deep neural networks for tagging of highly energetic jets resulting from top quark decays has focused on image based techniques or multivariate approaches using high-level jet substructure variables. Here, a sequential approach to this task is taken by using an ordered sequence of jet constituents …

We test the price momentum effect in the Korean stock markets under the momentum universe shrinkage to subuniverses of the KOSPI 200. Performance of the momentum strategy is not homogeneous with respect to change of the momentum universe. It is found that some submarkets generate the higher momentum returns than other …

Introduces homotopy momentum sections on multisymplectic manifolds.

MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.

Customer momentum is a positive relationship between a firm's returns and past returns of its customers.

Massless Majorana spinors cannot exist in Kerr spacetime under certain conditions.

AdEMAMix optimizer improves model performance and convergence speed.

The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.

This paper examines momentum spillover across multiple asset classes using only pricing data.

The paper analyzes how hyperparameters affect SGD with momentum's convergence rate.