New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
Study on test risk dynamics in learning theory with stochastic gradient flow.
problem Understanding test risk in stochastic gradient flow dynamics.
method Path integral formulation for small learning rates, explicit computation for weak features.
result Explicit corrections due to stochastic term in dynamics, good agreement with simulations.
Improved sampling efficiency with Random Reshuffling for Langevin dynamics.
problem Sampling efficiency in stochastic gradient algorithms.
method Random Reshuffling for Stochastic Gradient Langevin Dynamics (SGLD).
result Random Reshuffling leads to improved performance in sampling.
New method corrects bias in stochastic gradient samplers.
problem Bias in stochastic gradient samplers.
method Gradient-Guided Monte Carlo with stochastic gradients.
result Corrected sampler yields nonzero acceptance probabilities.
Many recent Markov chain Monte Carlo (MCMC) samplers leverage continuous dynamics to define a transition kernel that efficiently explores a target distribution. In tandem, a focus has been on devising scalable variants that subsample the data and use stochastic gradients in place of full-data gradients in the dynamic s…
FSGLD improves federated data sampling by correcting noisy gradients.
problem Noisy gradients and delayed communication in federated data.
method Conductive gradients to correct noisy gradients in distributed SGLD.
result FSGLD converges to true posterior even with delayed communication.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
SGBD algorithm improves robustness in Bayesian sampling.
problem Inefficiency of existing MCMC algorithms in large datasets.
method Extends Barker MCMC to stochastic gradient framework, introducing bias-corrected version.
result SGBD is more robust to hyperparameter tuning and gradient noise.
Paper approximates risk measures using SGD with Langevin dynamics.
problem Approximating arbitrary law invariant risk measures.
method Stochastic Gradient Langevin Dynamics (SGD-Langevin) for general risk measures.
result Non-asymptotic convergence rates of the approximation algorithm.
Paper improves convergence rate of Langevin Dynamics algorithms.
problem Sampling problems and non-convex optimization in machine learning.
method Stochastic Variance Reduced Gradient Langevin Dynamics and Stochastic Recursive Gradient Langevin Dynamics with improved convergence rates.
result Proves convergence to objective distribution under weaker conditions.
ICSGLD improves efficiency in posterior sampling for big data.
problem Efficient posterior sampling for large datasets.
method Embarrassingly parallel multiple-chain CSGLD with efficient interactions.
result ICSGLD is more efficient than a single-chain CSGLD.
Analysis of SGD for Gaussian mixture classification using dynamical mean-field theory.
problem Learning dynamics of SGD for a neural network classifying Gaussian mixture.
method Applying dynamical mean-field theory to track SGD dynamics in high dimensions.
result Reveals how SGD navigates the non-convex loss landscape.
We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood math…
SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.
Improved privacy analysis for stochastic gradient descent.
problem Analyzing privacy leakage in noisy stochastic gradient descent.
method Modeling Rényi divergence dynamics with Langevin diffusions, proving exponential privacy loss convergence for smooth and strongly convex objectives.
result Privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size.
We present a unified framework to analyze the global convergence of Langevin dynamics based algorithms for nonconvex finite-sum optimization with n component functions. At the core of our analysis is a direct analysis of the ergodicity of the numerical approximations to Langevin dynamics, which leads to faster conver…
Enhances SGLD for log-concave posteriors with asynchronous computation.
problem Sampling log-concave posterior distributions efficiently.
method Integrates asynchronous computation into SGLD with delayed gradients.
result Convergence in measure is not significantly affected by delayed gradient information.
Simplified SGD interpretation as Ito process for broader applicability.
problem Lack of generality in current SGD interpretation.
method Introduced a simplified scheme for discrete-time approximation of Ito process.
result Flexibly interprets SGD and SGLD, providing insights into their asymptotic properties.
Adaptive algorithm improves convergence rate of Langevin dynamics.
problem Improving convergence rate of Langevin dynamics.
method Adaptive non-reversible stochastic gradient Langevin dynamics algorithm.
result Improved convergence rate of the algorithm.
Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high vari…
The paper analyzes variance reduction in stochastic gradient Langevin dynamics.
problem Reducing the variance of stochastic gradient estimators in Langevin dynamics.
method Central limit theorem and Poisson equation analysis for variance characterization.
result Anti-symmetric perturbations can reduce the variance of non-reversible Langevin dynamics.
Unified bounds for random subset generalization error and improved SGD Langevin dynamics.
problem Generalization error bounds for random subsets and stochastic gradient Langevin dynamics.
method Unified framework based on Hellström and Durisi's work, extending bounds for Langevin dynamics.
result Unified and refined bounds for generalization error in stochastic gradient Langevin dynamics.
Analyzes high-dimensional SGD dynamics using DMFT.
problem Understanding the high-dimensional behavior of multi-pass SGD with small batch sizes.
method Derives DMFT equations for high-dimensional SGD dynamics.
result Proves DMFT equations characterize the asymptotic distribution of SGF parameters.
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.
New analysis for sampling from non-convex distributions with dependent data.
problem Sampling from non-logconcave distributions in stochastic optimization.
method Stochastic Gradient Langevin Dynamics (SGLD) with dependent data streams.
result Sharper and uniform convergence estimates in L1-Wasserstein distance. Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better char…
NSGLD improves SGLD for non-convex optimization problems.
problem Optimizing non-convex objectives efficiently.
method Introducing non-reversible SGLD by adding an anti-symmetric matrix to the drift term of the Langevin diffusion.
result NSGLD converges faster to the same stationary distribution with non-asymptotic guarantees.
A new algorithm flattens multi-modal distributions for better deep learning.
problem Bayesian learning in big data with multi-modal distributions.
method Contour Stochastic Gradient Langevin Dynamics (CSGLD) algorithm.
result The CSGLD algorithm avoids local traps in deep neural networks.
Improved convergence for non-log-concave sampling.
problem Sampling from non-log-concave distributions.
method Novel conductance analysis of SGLD with auxiliary Markov Chain.
result SGLD achieves ε-sampling error with fewer evaluations.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
Improved convergence rates for MFLD in various gradient estimators.
problem Proving convergence rates for mean-field Langevin dynamics with stochastic gradient updates.
method General framework for propagation of chaos, including finite-particle approximation, time-discretization, and stochastic gradient approximation.
result Improved convergence rates for SGD and SVRG settings.
A new algorithm improves Bayesian federated learning by reducing communication overhead.
problem Bayesian federated learning constraints, including privacy, data ownership, and communication overhead.
method Proposes Quantised Langevin Stochastic Dynamics (QLSD) for Bayesian federated learning, using gradient compression and variance reduction techniques.
result Non-asymptotic and asymptotic convergence guarantees for QLSD and its improved versions.
New method efficiently computes gradients for stochastic differential equations.
problem Computing gradients for stochastic differential equations efficiently.
method Generalized adjoint sensitivity method to stochastic differential equations.
result Time-efficient and memory-efficient computation of gradients with high-order solvers.
The paper analyzes SGD in high-dimensional networks, revealing new scaling limits.
problem Understanding SGD dynamics in high-dimensional networks.
method Analyzing the effective dynamics of SGD using recent work on the subject.
result A new correction term emerges at the critical scaling regime, changing the phase diagram.
A new method for stochastic optimization using virtual gradients.
problem Stochastic optimization challenges in computational efficiency and memory usage.
method Inspired by dynamic programming, SVGD uses a computational graph and automatic differentiation for efficient optimization.
result Experimental results show SVGD outperforms other methods on multiple datasets and network models.
Conservative SPDEs emerge from fluctuating SGD dynamics in neural networks.
problem Understanding the convergence of stochastic gradient descent to SPDEs.
method Mean-field analysis and central limit theorem for SPDEs.
result Optimal convergence rates for SPDEs derived from SGD.
Proves equations for high-dimensional gradient-based methods from Gaussian data.
problem High-dimensional asymptotics of gradient-based learning algorithms.
method Closed-form equations derived from dynamical mean-field theory.
result Equations match those from discretized DMFT for gradient flow.
New rates for GLD and SGLD in infinite-dimensional spaces without dimensionality issues.
problem Gradient Langevin dynamics and SGLD convergence rates in high-dimensional spaces.
method Analysis of GLD and SGLD in infinite-dimensional Hilbert spaces, using stochastic differential equations and Markov chains.
result Derivation of dimension-free convergence rates for GLD and SGLD.
Adaptively preconditions SGLD for faster convergence and better generalization.
problem Pathological curvature in deep network loss landscapes.
method Adaptive estimation of noise parameters to precondition isotropic gradient noise.
result Adaptively preconditioned SGLD achieves faster convergence and generalization equivalent of SGD.
A new method reduces variance in SGMCMC by preferentially subsampling data.
problem High variance in stochastic gradient estimates impacts sampler performance.
method Use a non-uniform probability distribution to preferentially subsample data points and adaptively adjust subsample size.
result Maintains accuracy while substantially reducing average subsample size.
SGLRW improves robustness of stochastic gradient MCMC methods.
problem Sensitivity to minibatch size and gradient noise in stochastic-gradient MCMC methods.
method Proposes Stochastic Gradient Lattice Random Walk (SGLRW) with lattice-based discretization.
result SGLRW remains stable in regimes where SGLD fails, including heavy-tailed gradient noise.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
Paper proposes Langevin dynamics for adaptive IRL of stochastic gradient algorithms.
problem Estimating reward functions from noisy gradient estimates of stochastic gradient agents.
method Generalized Langevin dynamics algorithm for IRL.
result Proposed algorithms asymptotically generate samples proportional to exp(R(θ)).
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
New method improves sampling efficiency in complex stochastic systems.
problem Sampling efficiency in nonconvex stochastic gradient cases.
method Reflection coupling for unadjusted generalized Hamiltonian Monte Carlo.
result Quantitative Gaussian concentration bounds and convergence rates established.
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Develops RL for dynamic risk assessment in stochastic optimization.
problem Time-consistent risk assessment in stochastic optimization problems.
method Model-free reinforcement learning with dynamic convex risk measures, time-consistent dynamic programming, policy gradient updates, actor-critic neural network optimization.
result Demonstrates optimal policies for statistical arbitrage, financial hedging, and robot control.