New model explains deep learning performance at large learning rates.
problem Understanding deep learning performance at different learning rates.
method Developed neural networks with solvable training dynamics.
result Large learning rates lead to convergence to flatter minima.
This paper demonstrates dynamic hyper-parameter setting, for deep neural network training, using Mutual Information (MI). The specific hyper-parameter studied in this paper is the learning rate. MI between the output layer and true outcomes is used to dynamically set the learning rate of the network through the trainin…
New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In …
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
OMD and DA perform similarly in static settings but OMD is inferior under dynamic learning rates.
problem Proving and understanding the performance difference between OMD and DA under dynamic learning rates.
method Introducing stabilization to OMD and modifying its convergence analysis.
result OMD with stabilization and DA have the same performance guarantees under dynamic learning rates.
Study reveals dynamics of neural networks with normalization, weight decay, and SGD.
problem Understanding the equilibrium condition in Spherical Motion Dynamics (SMD).
method Investigates SMD by exploring the cause of equilibrium condition, introducing assumptions, proposing angular update, and verifying theoretical results.
result Proves weight norm and angular update can converge at linear rate under given assumptions.
Adaptive algorithm improves convergence rate of Langevin dynamics.
problem Improving convergence rate of Langevin dynamics.
method Adaptive non-reversible stochastic gradient Langevin dynamics algorithm.
result Improved convergence rate of the algorithm.
Transfer learning improves chaotic dynamics predictions with less data.
problem Efficiently predicting chaotic dynamics with limited data.
method Transfer learning for nonlinear dynamics, optimizing transfer rate and leveraging small-scale turbulence universality.
result Significantly more accurate inference of chaotic dynamics achieved.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
Bayesian optimisation for dynamically adjusting learning rates in machine learning models.
problem Dynamic adjustment of learning rates schedules in machine learning models.
method Probabilistic model based on latent Gaussian processes and auto-/regressive formulation.
result Flexibly adjusts learning rates schedules to abrupt changes of behaviours.
Study learns dynamics of linear systems from multiple short trajectories.
problem Learning dynamics of autonomous linear systems from multiple short trajectories.
method Finite sample analysis for stable and unstable systems, adjusting trajectory length for marginally stable systems.
result Learning rate of O ( 1 N ) \mathcal{O}(\frac{1}{\sqrt{N}}) O ( N 1 ) for both stable and unstable systems. A new dynamic learning-rate scheme for optimization.
problem Manual tuning of learning rates in optimization is time-consuming and error-prone.
method Locally optimal stepsize estimation for dynamic learning-rate scheduling.
result Our method achieves comparable performance with minimal tuning.
New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
Study minimax rates for online learning with time-varying dynamics.
problem Online learning with time-varying state and cost dynamics.
method Non-constructive upper and lower bounds, complexity and stability terms.
result Characterization of minimax rates and necessary conditions for learnability.
GOALS improves learning rate selection for dynamic MBSS in deep learning.
problem Challenges in selecting learning rates for dynamic MBSS in deep learning.
method Gradient-only approximation line search (GOALS) for dynamic MBSS loss functions.
result GOALS reduces model errors in multimodal cases.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
Analyzes symmetries in neural networks to predict learning dynamics.
problem Understanding the dynamics of neural network parameters during training.
method Unified theoretical framework based on symmetries and conservation laws.
result Symmetries impose geometric constraints on gradients and Hessians, leading to conservation laws.
DynBRO learns robustly from dynamic Byzantine workers.
problem Fault-tolerant distributed learning with dynamic Byzantine workers.
method Multi-level Monte Carlo (MLMC) gradient estimation and adaptive learning rate.
result DynaBRO nearly matches static setting's convergence rate with O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) Byzantine worker changes. Investigates optimal portfolio strategies in markets with latent side information.
problem Investment problem in markets with latent dependence structure and side information.
method Dynamic and constant portfolio strategies, analyzing log-optimal portfolio as benchmark.
result Optimal dynamic strategy growth rate asymptotically converges to constant strategy in stationary markets.
New algorithm reduces adaptation lag in online model selection.
problem Adaptation lag in online model selection for non-stationary environments.
method Optimistic online mirror descent with safeguarded large learning rates.
result Reduces adaptation lag from hundreds of rounds to a few rounds.
New insights into how large learning rates affect transformer training dynamics.
problem Understanding how large learning rates impact the training of transformer models.
method Analyzing a simplified linear transformer model with a two-factor product map.
result Large learning rates can lead to various training outcomes including cycles, chaos, or divergence.
The dynamics of minority games with agents trading on different time scales is studied via dynamical mean-field theory. We analyze the case where the agents' decision-making process is deterministic and its stochastic generalization with finite heterogeneous learning rates. In each case, we characterize the macroscopic…
Safe learning of stochastic dynamics with safety constraints.
problem Learning controlled stochastic dynamics with safety constraints.
method Iterative expansion of a safe control set using kernel-based confidence bounds.
result The method ensures safe exploration and efficient estimation of system dynamics.
We formulate and study a general family of (continuous-time) stochastic dynamics for accelerated first-order minimization of smooth convex functions. Building on an averaging formulation of accelerated mirror descent, we propose a stochastic variant in which the gradient is contaminated by noise, and study the resultin…
The paper models SOFR and EFFR dynamics, reconciling diffusive and piecewise paths.
problem Updating interest rate models for SOFR, which is becoming a key benchmark.
method Calibrates a model to SOFR and EFFR futures prices, reconciling diffusive and piecewise paths.
result The model reflects key empirical features of SOFR dynamics and reconciles diffusive and piecewise paths.
A robust machine learning approach forecasts U.S. Treasury yields, reducing risk for investors.
problem Noisy and uncertain U.S. Treasury yields pose risk to forecast users.
method Formulates yield curve forecasting as a distributionally robust problem, combining factor models and machine learning.
result Robust forecast combinations improve out-of-sample performance across different maturity periods.
We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rate, batch size and the properties of the final minima, such as width or generalization, remains an open question. In order to tackle this pro…
Study on reinforcement learning dynamics using statistical physics.
problem Understanding how reinforcement learning dynamics interact with parameters and state features.
method Statistical physics concepts applied to temporal difference learning with linear function approximators.
result Stochastic semi-gradient noise leads to significant plateaus in value error.
Analyzes optimal learning rate schedules in high-dimensional non-convex optimization problems.
problem Optimizing high-dimensional non-convex loss landscapes.
method Langevin optimization with learning rate decaying as \(η(t) = t^{-β}\).
result To speed up optimization without getting stuck in saddles, a decay rate \(β < 1\) is optimal, contrary to convex setups where \(β = 1\).
SGD works well with large learning rates at the edge of stability.
problem Stochasticity at the edge of stability in deep learning.
method Sharp convergence guarantees for SGD with multiclass cross-entropy loss.
result SGD self-stabilizes, ensuring convergence with large learning rates.
New CVs preserve transition rates in molecular dynamics.
problem Designing CVs that accurately capture rare events in high-dimensional systems.
method Integrating manifold learning and group-invariant featurization to construct neural network-based CVs that satisfy orthogonality conditions.
result Achieved a CV for butane that reproduces the anti-gauche transition rate with less than ten percent relative error.
ISOKANN learns collective variables and effective dynamics for metastable transitions.
problem Understanding metastable transitions in complex molecular systems.
method Integrates Koopman operators with neural networks to extract CVs and effective dynamics.
result Reconstructs coarse-grained kinetics and reproduces transition times across barriers.
The paper analyzes how GANs converge using dual metric flows.
problem Understanding the convergence dynamics of GANs.
method Investigates the convergence of GANs using dual metric flows, formal definitions, and proving convergence.
result GAN learning dynamics converge to a limit when learning rate is small.
Study the properties of SGD in non-vanishing learning rate regime.
problem Understanding the noise and fluctuation in SGD with finite learning rates.
method Derive exact solvable results for discrete-time SGD in quadratic loss functions.
result Fluctuation caused by discrete-time dynamics is larger than continuous-time theory predicts.
Analyzes Indian commercial dynamism using time series data.
problem Understanding commercial dynamism in India.
method Time series analysis of various economic indicators.
result Detailed insights into growth rate, trade balance, etc.
An Entropic Dynamics of exchange rates is laid down to model the dynamics of foreign exchange rates, FX, and European Options on FX. The main objective is to represent an alternative framework to model dynamics. Entropic inference is an inductive inference framework equipped with proper tools to handle situations where…
Matrix factorization is a key component of collaborative filtering-based recommendation systems because it allows us to complete sparse user-by-item ratings matrices under a low-rank assumption that encodes the belief that similar users give similar ratings and that similar items garner similar ratings. This paradigm h…
The paper offers error bounds for quantized dynamical models.
problem Accuracy of dynamical models from dependent data sequences.
method Developed uniform error bounds for quantized models and imperfect optimization algorithms.
result Unified bounds for slow and fast rates, scaling with model encoding bits.
We discover scaling laws for kernel regression loss under various learning rate schedules.
problem Understanding loss dynamics and learning rate schedules in kernel regression.
method Theoretical analysis of stochastic gradient descent on a power-law kernel regression model.
result Established a Functional Scaling Law (FSL) capturing the full loss trajectory under arbitrary learning rate schedules.
Algorithm learns linear systems from partial observations with near-optimal rate.
problem Identifying linear dynamical systems from partial observations, especially those with long-term memory.
method Multi-scale low-rank approximation using SVD on Hankel matrices of increasing sizes, combined with Fourier domain concentration bounds.
result Near-optimal rate of $\widetilde O\left(\sqrt\frac{d}{T}
ight)$ in H 2 \mathcal{H}_2 H 2 error, with logarithmic dependence on memory length. Negative momentum accelerates convergence in minimax games but at a suboptimal rate.
problem The convergence rate of negative momentum in minimax games is suboptimal.
method Extending variational inequality formulation, connecting momentum method with Chebyshev polynomials.
result Negative momentum accelerates convergence locally but at a suboptimal rate.
Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Rec…
Machine learning aids excited-state molecular dynamics studies.
problem Challenges in studying electronically excited states of molecules.
method Employing machine learning techniques for excited-state molecular dynamics.
result Highlight successes and challenges in machine learning for excited-state processes.
GALA adapts learning rates online by aligning gradients, improving deep learning model performance.
problem Fine-tuning learning rates for deep learning models requires extensive grid search.
method GALA dynamically adjusts learning rates by tracking gradient alignment and local curvature.
result GALA produces a flexible, adaptive learning rate schedule that increases when gradients align.
Neural ARFIMA model improves exchange rate forecasting for BRIC economies.
problem Forecasting exchange rates for emerging markets with long-term memory and nonlinear dynamics.
method Integrates ARFIMA for long-memory with neural networks for nonlinear approximation.
result NARFIMA model outperforms benchmarks in BRIC exchange rate forecasting.
New SDE model from machine learning optimization with unique stationary distribution.
problem Stationary distribution of machine learning optimization models.
method Proved ergodicity and unique stationary distribution of power-law dynamic SDE.
result Power-law dynamic has a unique stationary distribution and is ergodic.