A new method reduces communication in distributed learning by 32x.
problem High communication costs in distributed machine learning.
method Distributed compressed SGD with Nesterov's momentum and blockwise compression.
result Achieves 32x reduction in communication cost and converges as fast as full-precision SGD.
Proposes blockwise adaptive stepsize for faster training and better generalization in deep learning.
problem Widespread use of coordinate-wise adaptive methods like RMSprop and Adam leads to worse generalization than SGD.
method Splits network parameters into blocks and uses a blockwise adaptive stepsize, balancing adaptivity and generalization.
result Blockwise adaptive gradient descent converges faster and has lower generalization error than coordinate-wise adaptive methods.
Blockwise bootstrap improves ASR performance testing for correlated data.
problem Testing reliability of WER improvements between ASR systems.
method Divide evaluation utterances into nonoverlapping blocks and resample these blocks.
result The variance estimator of absolute WER difference is consistent under mild conditions.
Paper addresses high-dimensional linear regression with missing data, proposing efficient and nearly unbiased estimators.
problem High-dimensional linear regression with blockwise missing covariates and partially observed responses.
method Proposes a computationally efficient estimator and nearly unbiased debiased estimators using blockwise imputation and estimating equations.
result Asymptotically valid confidence intervals and statistical tests constructed based on debiased estimators.
New method handles missing data using AI for efficient inference.
problem Parameter estimation and inference with blockwise missing data.
method Tractable solution using AI models and semiparametric theory.
result IBM(RAY) and IBM(Adaptive) estimators achieve efficiency gains.
Develops a gradient flow for Muon optimizer, a method for optimization.
problem Optimization of complex systems with matrix-valued parameters.
method Gradient flow on probability measures induced by regularized Muon optimizer.
result Derives continuous-time limits and proves Hamiltonian dissipation.
In this paper we purpose a blockwise descent algorithm for group-penalized multiresponse regression. Using a quasi-newton framework we extend this to group-penalized multinomial regression. We give a publicly available implementation for these in R, and compare the speed of this algorithm to a competing algorithm --- w…
Graphical lasso models ASR utterance dependencies for consistent WER estimation.
problem Modeling dependent structure among ASR utterances for accurate significance analysis.
method Graphical lasso for dependency modeling, followed by blockwise bootstrap resampling.
result Statistically consistent variance estimator of WER under mild conditions.
Mixture-of-experts (MoE) models are a powerful paradigm for modeling of data arising from complex data generating processes (DGPs). In this article, we demonstrate how different MoE models can be constructed to approximate the underlying DGPs of arbitrary types of data. Due to the probabilistic nature of MoE models, we…
Paper finds sharpness differences in transformer blocks accelerating LLM training.
problem Understanding and accelerating large language model pre-training.
method Uncovering sharpness disparity across transformer blocks and proposing Blockwise Learning Rate.
result Blockwise Learning Rate strategy accelerates LLM pre-training with lower loss and speedup.
MSB framework improves survival prediction in immunotherapy patients with missing data.
problem High dimensionality and blockwise missingness in multimodal clinical data.
method MSB is a late-fusion framework that independently models modality-specific features before aggregating predictions via cross-validated stacking.
result MSB outperformed baseline algorithms in predicting progression-free survival in lung cancer patients.
Novel tensor perturbation bounds for orthogonal iteration methods.
problem Developing robust bounds for tensor reconstruction and subspace estimation.
method Blockwise tensor perturbation bounds for high-order orthogonal iteration (HOOI).
result Upper bounds for singular subspace estimation converge linearly and tensor reconstruction error bound is characterized by a simple quantity.
Unified approach for learning quantum operations from measurements.
problem Accurate reconstruction of unknown quantum operations from noisy measurements.
method Matrix sensing techniques, randomized measurement design, blockwise measurement design, alternating least squares (ALS).
result The proposed method provides theoretical guarantees for the identifiability and recovery of low-rank superoperators in the presence of noise.
Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the le…
A framework estimates multiple precision matrices with shared structures.
problem Estimating multiple precision matrices with shared structures.
method Penalized likelihood framework with iterative algorithm alternating between convex and clustering problems.
result The method outperforms competitors and performs similarly to methods using prior information.
The study finds that factor momentum is significant only at short lags compared to stock momentum.
problem Investigating the relationship between factor momentum and stock momentum.
method Replicated earlier findings and conducted a spanning test controlling for stock momentum and factor exposure.
result Factor momentum is significant only at short lags after controlling for stock momentum and factor exposure.
Improved TD learning for non-i.i.d. Markovian data.
problem Convergence analysis of two time-scale TD learning under Markovian samples.
method Non-asymptotic convergence analysis of two time-scale TD with gradient correction under Markovian data.
result Two time-scale TD can converge as fast as O(log t/(t^(2/3))) under diminishing stepsize.
We test the price momentum effect in the Korean stock markets under the momentum universe shrinkage to subuniverses of the KOSPI 200. Performance of the momentum strategy is not homogeneous with respect to change of the momentum universe. It is found that some submarkets generate the higher momentum returns than other …
Introduces homotopy momentum sections on multisymplectic manifolds.
problem No specific problem stated; focuses on introducing a new concept.
method Introduces a new concept of homotopy momentum sections on multisymplectic manifolds.
result Shows that a gauged nonlinear sigma model with Wess-Zumino term has homotopy momentum section structure.
Customer momentum is a positive relationship between a firm's returns and past returns of its customers.
problem Understanding the relationship between a firm's returns and its customers' past returns.
method Examined customer momentum using a long-short equally-weighted decile portfolio and Fama-French factor models.
result Customer momentum generates significant monthly returns and is statistically significant.
Imputation-Powered Inference improves subpopulation efficiency in missing data settings.
problem Complex missing data patterns challenge standard inference methods.
method Imputation-Powered Inference (IPI) combines blackbox imputation with bias correction.
result IPI provides valid and efficient M-estimation under MCAR blockwise missingness.
This paper examines momentum spillover across multiple asset classes using only pricing data.
problem Challenges in studying momentum spillover across diverse asset classes due to lack of common characteristics.
method Utilised a linear and interpretable graph learning model to reveal momentum spillover network.
result Network momentum strategy yields a Sharpe ratio of 1.5 and an annual return of 22%.
The paper analyzes how hyperparameters affect SGD with momentum's convergence rate.
problem The role of hyperparameters in SGD with momentum's convergence rate.
method Theoretical analysis using a hyperparameters-dependent stochastic differential equation (hp-dependent SDE).
result The optimal linear rate of convergence depends on both the learning rate and the momentum coefficient.
New model captures time series dependence across and within blocks.
problem Complex multivariate time series dependence structures.
method Time series Gaussian chain graph models with directed and undirected edges.
result Consistent recovery of time series chain graph structure.
This paper presents generalized momentum mappings for covariant Hamiltonian field theories. The new momentum mappings arise from a generalization of symplectic geometry to LVY, the bundle of vertically adapted linear frames over the bundle of field configurations Y. Specifically, the generalized field momentum obs…
We give a detailed discussion about existence and uniqueness of Lu's momentum map. More precisely, we introduce the infinitesimal momentum map, and we study its properties. This allows us to describe the theory of reconstruction of the momentum map from the infinitesimal one. We provide the conditions for the uniquenes…
Momentum ResNets improve ResNets' memory efficiency.
problem Memory inefficiency in deep residual neural networks (ResNets).
method Adding a momentum term to the forward rule of ResNets to make them invertible.
result Momentum ResNets can learn any linear mapping up to a multiplicative factor, improving memory efficiency.
The paper analyzes how momentum affects convergence in stochastic gradient methods.
problem Lack of clear understanding of momentum's impact on convergence and performance.
method Unified analysis of several popular algorithms using the QHM formulation.
result Provides practical guidelines for setting learning rate and momentum parameters.
We introduce various quantitative and mathematical definitions for price momentum of financial instruments. The price momentum is quantified with velocity and mass concepts originated from the momentum in physics. By using the physical momentum of price as a selection criterion, the weekly contrarian strategies are imp…
DEAM optimizes momentum weights dynamically to improve deep learning model training.
problem Errors in momentum weights propagate errors in optimization algorithms like ADAM.
method DEAM computes adaptive momentum weights based on discriminative angles, reducing hyperparameters and introducing a backtrack term.
result DEAM achieves faster convergence rates in both convex and non-convex deep learning model training.
Adapting momentum from optimization to reinforcement learning.
problem Improving the convergence and stability of reinforcement learning algorithms.
method Introducing Momentum Value Iteration (MoVI) by incorporating an average of consecutive state-action value functions, inspired by the concept of momentum in optimization.
result MoVI improves the convergence and stability of reinforcement learning algorithms, as demonstrated by experiments on Atari games.
Sparse learning speeds up neural network training without sacrificing accuracy.
problem Training deep neural networks efficiently while maintaining performance.
method Sparse momentum algorithm that redistributes and grows weights based on momentum magnitude.
result State-of-the-art sparse performance on various datasets with up to 5.61x faster training.
New algorithm Momentum-QNG improves optimization of quantum circuits.
problem Optimizing variational quantum circuits to avoid local minima.
method Applied Langevin dynamics to QNG, introducing momentum term.
result Momentum-QNG outperforms basic QNG and other optimizers.
Momentum speeds up evolutionary processes in machine learning.
problem Accelerating convergence in evolutionary dynamics.
method Combining momentum from machine learning with evolutionary dynamics using information divergences as Lyapunov functions.
result Momentum accelerates convergence of evolutionary dynamics, including the replicator equation and Euclidean gradient descent.
Contact manifolds' momentum polytopes are convex.
problem Understanding the structure of contact manifolds.
method Using isomorphism to toric varieties.
result Momentum polytopes of contact manifolds are convex.
Unified model learns from both time-series and cross-sectional momentum features.
problem Separate time-series and cross-sectional momentum strategies do not consider concurrent relationships.
method Spatio-Temporal Momentum strategies using neural networks to combine both types of momentum.
result Simple neural network with single fully connected layer generates trading signals for all assets.
SMG combines shuffling and momentum for non-convex optimization.
problem Non-convex finite-sum optimization problems.
method Shuffling Gradient-based method with momentum.
result Established state-of-the-art convergence rates for SMG.
GMC uses global momentum for sparse communication in distributed learning.
problem Latency and bandwidth limitations in network communication for large-scale deep learning models.
method GMC utilizes global momentum for sparse communication in DMSGD, improving convergence and accuracy.
result GMC and GMC+ achieve higher test accuracy and faster convergence compared to local momentum methods.
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
One has not any conventional energy-momentum conservation law in Lagrangian field theory, but relations involving different stress-energy-momentum tensors associated with different connections. It is not obvious how to choose the true energy-momentum tensor. This problem is solved in the framework of the multimomentum …
The paper extends a theorem about momentum maps to singular symplectic spaces.
problem Extending a theorem about momentum maps to singular symplectic spaces.
method Using integral affine stratification and equivariant locally trivial fibrations, the paper extends the linear variation theorem to singular values of the momentum map.
result Cohomology classes of symplectic forms on reduced spaces vary linearly within strata.
This paper analyzes momentum Q-learning with finite-sample guarantees.
problem Improving Q-learning performance with momentum schemes.
method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.
result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.
Momentum improves deep learning generalization by stabilizing noise and learning features.
problem Improving generalization in deep learning models.
method Empirical and theoretical analysis of gradient descent with momentum (GD+M) vs. gradient descent (GD) in binary classification tasks.
result GD+M outperforms GD in generalization, especially in datasets with shared features and varying margins.
Generalizes momentum map to Courant algebroid for constrained mechanics.
problem Generalizing momentum map to new geometric structures.
method Generalized momentum section on Lie algebroid to Courant algebroid, constructed cohomological formulations.
result Identified momentum section in constrained Hamiltonian mechanics with Courant algebroid symmetry.
Momentum SGD fails to track nonstationary optima due to drift amplification.
problem Tracking nonstationary optima in stochastic optimization.
method Theoretical analysis of SGD and momentum variants under strong convexity and smoothness.
result Momentum incurs a drift-amplification penalty that diverges as the momentum parameter approaches 1, leading to systematic lag.
This paper concentrates on the time series momentum or contrarian effects in the Chinese stock market. We evaluate the performance of the time series momentum strategy applied to major stock indices in mainland China and explore the relation between the performance of time series momentum strategies and some firm-speci…
A new accelerated method with simpler momentum update rules.
problem Optimizing parameters in machine learning models.
method Proposes a novel accelerated stochastic gradient method with simpler momentum update rules.
result The method outperforms Sgdm and Adam in practical problems.