Conservative SPDEs emerge from fluctuating SGD dynamics in neural networks.
problem Understanding the convergence of stochastic gradient descent to SPDEs.
method Mean-field analysis and central limit theorem for SPDEs.
result Optimal convergence rates for SPDEs derived from SGD.
Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.
problem Long-term behavior of stochastic gradient descent in non-smooth settings.
method Functional central limit theorem applied to rescaled trajectory of SGD.
result Characterization of long-term fluctuations around the minimizer.
Proposes a new adaptive gradient method based on gradient differences.
problem Manual tuning of stepsize in vanilla gradient methods.
method Adaptation driven by cumulative squared norms of gradient differences.
result More robust than AdaGrad in various settings.
New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
Derives scaling limits and fluctuations for SGD in high dimensions.
problem Understanding SGD behavior in high-dimensional settings with varying noise levels.
method Interacting particle system approach, treating SGD iterates as such, with covariance structure considered.
result Precise three-step phase transition observed in SGD behavior: ballistic, diffusive, then random.
This work studies fluctuation in multilayer neural networks using mean field theory.
problem Understanding fluctuation in multilayer neural networks with mean field training.
method Developed a second-order mean field limit to capture fluctuation, demonstrating stability of gradient descent training.
result Gradient descent training in multilayer networks biases towards minimal fluctuation, even after convergence.
Study the properties of SGD in non-vanishing learning rate regime.
problem Understanding the noise and fluctuation in SGD with finite learning rates.
method Derive exact solvable results for discrete-time SGD in quadratic loss functions.
result Fluctuation caused by discrete-time dynamics is larger than continuous-time theory predicts.
Gradient descent dynamics in wide neural networks are analyzed using a dynamical CLT.
problem Understanding the fluctuations in wide shallow neural networks trained via gradient descent.
method Dynamical Central Limit Theorem (CLT) applied to neural network dynamics.
result Asymptotic fluctuations remain bounded in mean square throughout training.
The paper analyzes variance reduction in stochastic gradient Langevin dynamics.
problem Reducing the variance of stochastic gradient estimators in Langevin dynamics.
method Central limit theorem and Poisson equation analysis for variance characterization.
result Anti-symmetric perturbations can reduce the variance of non-reversible Langevin dynamics.
The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations t…
New method optimizes SDE models using continuous-time gradient descent.
problem Optimizing over the stationary distribution of SDE models.
method Continuous-time stochastic gradient descent for SDE models.
result Asymptotic convergence to the direction of steepest descent.
MSTGD optimizes gradient descent with stratified sampling for faster convergence.
problem Fluctuation in gradient expectation and variance between iterations.
method Memory Stochastic Stratified Gradient Descent (MSTGD) with stratified sampling and variance reduction.
result MSTGD achieves an exponential convergence rate independent of dataset size and batch size.
We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural net…
Paper introduces a new gradient statistic to improve deep learning convergence.
problem Fluctuation effect of gradient updates between iterations.
method Introduces an unbiased stratified statistic \(\bar{G}_{mst}\) and a new algorithm MSSG.
result MSSG algorithm outperforms other sgd-like algorithms in training deep models.
Noise causes learning plateaus in neural networks.
problem Plateau phenomena in online learning due to vanishing gradients.
method Analysis of stochastic gradient descent in multi-layer perceptrons.
result Noise induces synchronisation leading to strong plateaus.
Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.
problem Fluctuations in deep networks with Gaussian weights can impair training, especially in networks with depth comparable to width.
method Analytical and numerical studies of fully-connected networks with orthogonal weight initialization and tanh activations.
result Rectangular networks with orthogonal weights have stable fluctuations independent of network depth, leading to better generalization and training speed.
In this paper, we analyze Twitter signals as a medium for user sentiment to predict the price fluctuations of a small-cap alternative cryptocurrency called \emph{ZClassic}. We extracted tweets on an hourly basis for a period of 3.5 weeks, classifying each tweet as positive, neutral, or negative. We then compiled these …
Functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
problem Fluctuations of boosting processes around their deterministic limit
method Stochastic perturbation analysis of ODEs in Banach spaces
result Rescaled deviations converge to a Gaussian process
Study shows cryptocurrency price fluctuations become more similar to national currencies over time.
problem Understanding the volatility and inequality in cryptocurrency prices.
method Calculated inequality measures (Gini, Kolkata indices, Q factor) for cryptocurrency and national currency price fluctuations over 10 years. result Cryptocurrency price fluctuations become more similar to national currencies over time.
This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is able to efficiently optimize a centralized model under communication constraints. The optimizer achieves this by pushing a normalized sequence of first-order gradients to a pa…
Riemannian stochastic gradient descent approximates a diffusion process called Riemannian stochastic modified flow.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Using stochastic differential geometry, the paper shows RSGD can be approximated by the Riemannian stochastic modified flow (RSMF).
result RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process, increasing the order of approximation.
Bayesian models' singular fluctuation is shown to be akin to specific heat, influencing model complexity and generalization.
problem Understanding the thermodynamic interpretation of singular fluctuation in Bayesian models.
method Showed singular fluctuation as the curvature of Bayesian free energy and variance of log-likelihood observable under a Gibbs posterior.
result Singular fluctuation is the statistical analogue of specific heat, controlling model complexity and generalization.
We study the nature of fluctuations in variety of price indices involving companies listed on the New York Stock Exchange. The fluctuations at multiple scales are extracted through the use of wavelets belonging to Daubechies basis. The fact that these basis sets satisfy vanishing moments conditions makes them ideal to …
New spectral functionals for Dirac operators with inner fluctuations computed.
problem Spectral functionals and Dirac operators with inner fluctuations.
method Extension of spectral functionals for Dirac operators with inner fluctuations.
result Computed spectral Einstein functional for Dirac operator with inner fluctuations on even-dimensional spin manifolds.
We propose a new approach for properly analyzing stochastic time series by mapping the dynamics of time series fluctuations onto a suitable nonequilibrium surface-growth problem. In this framework, the fluctuation sampling time interval plays the role of time variable, whereas the physical time is treated as the analog…
The paper develops statistical inference for gradient flows in optimization.
problem Uncertainty quantification along the entire optimization path.
method Uniform central limit theorem and algorithm-aware covariance estimator.
result Asymptotically valid confidence intervals for target parameter.
The average economic agent is often used to model the dynamics of simple markets, based on the assumption that the dynamics of many agents can be averaged over in time and space. A popular idea that is based on this seemingly intuitive notion is to dampen electric power fluctuations from fluctuating sources (as e.g. wi…
We propose a new approach for analyzing price fluctuations in their strongly correlated regime ranging from minutes to months. This is done by employing a self-similarity assumption for the magnitude of coarse-grained price fluctuation or volatility. The existence of a Cramer function, the characteristic function for s…
We analyze daily prices of 29 commodities and 2449 stocks, each over a period of ≈15 years. We find that the price fluctuations for commodities have a significantly broader multifractal spectrum than for stocks. We also propose that multifractal properties of both stocks and commodities can be attributed mainl…
Trading affects grid frequency fluctuations, making them more extreme.
problem Impact of trading on grid frequency stability.
method Analysis of frequency time series from 2011 and 2017.
result Trading modifies frequency fluctuation statistics, making large deviations more likely.
A phenomenological investigation of the endogenous and exogenous dynamics in the fluctuations of capital fluxes is investigated on the Chinese stock market using mean-variance analysis, fluctuation analysis and their generalizations to higher orders. Non-universal dynamics have been found not only in α exponents diff…
Study identifies contagion in aggregated defaults despite environmental changes.
problem Identify contagion in aggregated default counts with fluctuating probabilities.
method Compare three contagion mechanisms (Davis-Lo, Torri, Vasicek) under i.i.d. and hierarchical specifications.
result Threshold contagion is largely absorbed into environmental heterogeneity, while cumulative contagion leaves a persistent signature.
In this paper we compare market price fluctuations with the response to fundamental price drops within the Lux-Marchesi model which is able to reproduce the most important stylized facts of real market data. Major differences can be observed between the decay of spontaneous fluctuations and of changes due to external p…
SAM improves generalization by operating near the edge of stability.
problem Improving generalization in neural networks.
method Sharpness-Aware Minimization (SAM) approach to training neural networks.
result SAM operates near the 'edge of stability' identified by the analysis.
We address the question of how stock prices respond to changes in demand. We quantify the relations between price change G over a time interval Δt and two different measures of demand fluctuations: (a) Φ, defined as the difference between the number of buyer-initiated and seller-initiated trades, and (b) Ω, def…
The paper analyzes SGD in high-dimensional networks, revealing new scaling limits.
problem Understanding SGD dynamics in high-dimensional networks.
method Analyzing the effective dynamics of SGD using recent work on the subject.
result A new correction term emerges at the critical scaling regime, changing the phase diagram.
Study on price fluctuations in NFT market, showing heavy-tailed distributions and long-range memory.
problem Characterizing price fluctuations in NFT market.
method Analysis of capitalization, floor price, transactions, inter-transaction times, and volume value of NFTs.
result NFT market exhibits heavy-tailed probability distribution functions, well described by stretched exponentials, with long-range memory.
Spectral clustering performance depends on eigenvector fluctuations, shown to be Gaussian.
problem Predicting the performance of spectral clustering.
method General spike random matrix model and rotational invariance of noise.
result Fluctuations of eigenvector entries are Gaussian in large-dimensional regime.
Inexact subgradient methods work well for semialgebraic functions with additive errors.
problem Approximate gradients in machine learning and optimization.
method Inexact subgradient methods with persistent additive errors in semialgebraic functions.
result Iterates eventually fluctuate near the critical set with a proximity of O(ερ), where ε is the magnitude of subgradient evaluation errors. We analyze the fluctuation of the loss from default around its large portfolio limit in a class of reduced-form models of correlated firm-by-firm default timing. We prove a weak convergence result for the fluctuation process and use it for developing a conditionally Gaussian approximation to the loss distribution. Nume…
In this manuscript we present a comprehensive study on the multifractal properties of high-frequency price fluctuations and instantaneous volatility of the equities that compose Dow Jones Industrial Average. The analysis consists about quantification of dependence and non-Gaussianity on the multifractal character of fi…
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
We constructed an analog electrical circuit which generates fluctuations in which probability density function has power law tails. In the circuit fluctuations with an arbitrary exponent of the power law can be obtained by adjusting the resistance. With this low cost circuit the random fluctuations which have the simil…
The study examines neural networks with random weights and biases, finding that depth-to-width ratio controls fluctuations and correlations.
problem Exploring the exploding and vanishing gradient problem in neural networks with random weights and biases.
method Sharp estimates of joint cumulants and solving cumulant recursions in powers of 1/n.
result The depth-to-width ratio L/n plays a crucial role in controlling fluctuations and correlations, leading to the occurrence of exploding and vanishing gradients. The financial market and turbulence have been broadly compared on account of the same quantitative methods and several common stylized facts they shared. In this paper, the She-Leveque (SL) hierarchy, proposed to explain the anomalous scaling exponents deviated from Kolmogorov monofractal scaling of the velocity fluctu…
Study uses neural networks to predict wall quantities in turbulent flows.
problem Predicting wall quantities in turbulent open channel flows.
method Training convolutional neural networks (FCN) and a proposed R-Net architecture to predict wall-shear-stress and wall pressure.
result R-Net architecture performs better and predicts wall quantities with around 10% error.
This paper proposes a framework to predict long-term trends and short-term fluctuations in multivariate time series.
problem Existing prediction methods often ignore the distinction between long-term trends and short-term fluctuations.
method The paper introduces a MTS forecasting framework that uses both original time series and its first difference to capture long-term trends and short-term fluctuations.
result The proposed method improves forecasting performance by using more supervision information.
Predicting absolute magnitude of fluctuations of price, even if their sign remains unknown, is important for risk analysis and for option prices. In the present work, we display our predictions about absolute magnitude of daily fluctuations of the Dow Jones Industrials Average (DJIA), utilizing the original theory of c…