Langevin algorithms improve training of very deep neural networks, especially for image classification.
problem Training very deep neural networks is challenging due to increased non-linearity and the risk of getting stuck in local minima.
method Comparison of Langevin and non-Langevin algorithms for training deep neural networks, introduction of Layer Langevin algorithm.
result Langevin algorithms, especially Layer Langevin, lead to significant improvements in training deep neural networks, particularly for image classification tasks.
SLMC improves sampling efficiency for high-dimensional distributions.
problem Sampling from high-dimensional distributions is computationally challenging.
method SLMC projects Langevin updates onto subsampled eigenblocks of a time-varying preconditioner.
result SLMC offers superior adaptability and computational efficiency compared to traditional methods.
Paper analyzes and accelerates Langevin Monte Carlo methods using large deviations theory.
problem High-dimensional sampling problems in machine learning.
method Unified approach using large deviations theory to study and accelerate Langevin dynamics variants.
result Efficiency of Langevin dynamics variants demonstrated through numerical experiments.
New algorithms improve sampling from complex distributions.
problem Sampling from complex probability distributions efficiently.
method Regime-switching Langevin dynamics and Monte Carlo algorithms.
result Convergence guarantees and iteration complexities provided.
Study non-asymptotic Langevin Monte Carlo for Gibbs distributions.
problem Sampling from Gibbs distributions with dissipative potentials.
method Langevin-type algorithms based on Liptser--Shiryaev theory and Poincaré inequalities.
result Upper bound on 2-Wasserstein distance for accurate approximation.
Unified approach for sampling non-differentiable and heavy-tailed targets.
problem Sampling non-differentiable and heavy-tailed distributions using Langevin algorithms.
method Anchored Langevin dynamics, which modifies the Langevin diffusion with a smooth reference potential and multiplicative scaling.
result Non-asymptotic guarantees in the 2-Wasserstein distance to the target distribution.
Novel geometry-informed irreversible perturbation accelerates Langevin dynamics convergence.
problem Accelerating convergence of Langevin dynamics for Bayesian computation.
method Geometry-informed irreversible perturbation of Riemannian manifold Langevin dynamics.
result Improves estimation performance over irreversible perturbations that ignore geometry.
New methods use transport maps to improve Langevin dynamics for sampling.
problem Sampling high-dimensional, non-Gaussian distributions efficiently.
method Apply transport maps to accelerate Langevin dynamics convergence.
result Discretized processes converge to target distribution with non-asymptotic bounds.
In this article we develop geometric versions of the classical Langevin equation on regular submanifolds in euclidean space in an easy, natural way and combine them with a bunch of applications. The equations are formulated as Stratonovich stochastic differential equations on manifolds. The first version of the geometr…
The paper analyzes convergence of Langevin dynamics with time-dependent metrics.
problem Analyzing convergence of Langevin dynamics with time-dependent metrics.
method Formulated a modified gradient flow of the Kullback-Leibler divergence, selected a time-dependent relative Fisher information functional, and developed a time-dependent Hessian matrix condition.
result Proved convergence conditions for various Langevin dynamics.
Langevin MCMC samples efficiently from Riemannian manifolds with geometric Euler-Murayama analysis.
problem Efficient sampling from Gibbs distributions on Riemannian manifolds.
method Geometric Langevin MCMC, discretization error bound, contraction guarantee for Langevin Diffusion.
result Langevin MCMC iterates converge to the target distribution after a number of steps proportional to the inverse square of the desired accuracy.
Unified bounds for random subset generalization error and improved SGD Langevin dynamics.
problem Generalization error bounds for random subsets and stochastic gradient Langevin dynamics.
method Unified framework based on Hellström and Durisi's work, extending bounds for Langevin dynamics.
result Unified and refined bounds for generalization error in stochastic gradient Langevin dynamics.
Error estimates found between SGD with momentum and Langevin diffusion.
problem Quantifying the difference between SGD with momentum and Langevin diffusion.
method Established error estimates using 1-Wasserstein and total variation distances.
result Quantitative error estimates between SGD with momentum and underdamped Langevin diffusion.
Paper proposes new Langevin samplers for sampling from log-concave distributions with superlinear gradient growth.
problem Sampling from log-concave distributions with superlinear gradient growth.
method Proposes two novel discretizations of kinetic Langevin SDEs, showing contractivity and log-Sobolev inequality.
result Establishes non-asymptotic bounds in 2-Wasserstein distance between sampled distributions and target measures.
New Langevin Monte Carlo algorithms for sampling from nonsmooth distributions.
problem Sampling from distributions with nonsmooth convex composite potentials.
method Leveraging Bregman--Moreau envelopes and proximal operators in mirror descent.
result Efficiency in sampling from nonsmooth distributions, extending existing methods.
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
First order discretizations of Langevin diffusion can achieve better generalization error with additional smoothness assumptions.
problem Analyzing generalization error for first order discretizations of Langevin diffusion.
method Providing a sufficient smoothness condition to show that first order methods can achieve arbitrarily runtime complexity for a given expected generalization error.
result First order methods can achieve arbitrarily runtime complexity with additional smoothness assumptions.
We study the underdamped Langevin diffusion when the log of the target distribution is smooth and strongly concave. We present a MCMC algorithm based on its discretization and show that it achieves ε \varepsilon ε error (in 2-Wasserstein distance) in O ( d / ε ) \mathcal{O}(\sqrt{d}/\varepsilon) O ( d / ε ) steps. This is a significant improv…
Extends Langevin dynamics for constrained domains.
problem Optimization of constrained probability measures.
method Mirror mean-field Langevin dynamics (MMFLD).
result Linear convergence guarantees and propagation of chaos results.
Langevin Dynamics speeds up mixing time with manifold hypothesis and multi-scale approach.
problem Langevin Dynamics struggles in high dimensions and nonconvex landscapes.
method Utilizes manifold hypothesis to reduce mixing time and employs multi-scale approach to improve image generation quality.
result Mixing time depends on intrinsic dimension rather than ambient dimension, significantly reducing computational complexity.
New algorithm improves sampling from constrained spaces.
problem Sampling from constrained spaces efficiently.
method Metropolis-adjusted Mirror Langevin algorithm.
result Unbiased sampling with improved mixing time.
NSGLD improves SGLD for non-convex optimization problems.
problem Optimizing non-convex objectives efficiently.
method Introducing non-reversible SGLD by adding an anti-symmetric matrix to the drift term of the Langevin diffusion.
result NSGLD converges faster to the same stationary distribution with non-asymptotic guarantees.
Replica exchange Langevin diffusion accelerates nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Replica exchange Langevin diffusion, discretization analysis.
result Replica exchange accelerates convergence to global minima.
The Langevin Algorithm's stationary distribution is shown to be sub-exponential or sub-Gaussian under certain conditions.
problem Understanding the properties of the Langevin Algorithm's stationary distribution.
method Analysis using a rotation-invariant moment generating function (Bessel function) to study the stationary dynamics of the Langevin Algorithm.
result Concentration results for the Langevin Algorithm's stationary distribution π η π_η π η are established, showing it is sub-exponential or sub-Gaussian under convex or strongly convex potential conditions. We present a unified framework to analyze the global convergence of Langevin dynamics based algorithms for nonconvex finite-sum optimization with n n n component functions. At the core of our analysis is a direct analysis of the ergodicity of the numerical approximations to Langevin dynamics, which leads to faster conver…
Study shows uniform-time chaos propagation in mean field Langevin dynamics.
problem Understanding the convergence of marginal distributions in mean field dynamics.
method Assumed functional convexity of energy, used L p L^p L p -convergence and Wasserstein metrics. result Uniform-in-time propagation of chaos proved in both L 2 L^2 L 2 -Wasserstein and relative entropy. New Langevin algorithms improve sampling efficiency in high dimensions.
problem Sampling from log-concave and smooth distributions in high dimensions.
method Combining splitting and accurate integration methods for P P P -th order Langevin dynamics. result LMC algorithms converge faster with better dimension dependence as P P P increases. New method controls bias in unadjusted Hamiltonian Monte Carlo and underdamped Langevin.
problem Bias in unadjusted Hamiltonian Monte Carlo and underdamped Langevin samplers.
method Delocalization of bias technique applied to these samplers.
result Control W 2 W_2 W 2 bias with O ( K ) O(\sqrt{K}) O ( K ) integration steps for high-dimensional distributions. A new method for sampling from complex distributions using Langevin samplers.
problem Sampling from unnormalized Boltzmann densities.
method Probability flow ODE derived from linear stochastic interpolants, employing Langevin samplers.
result Efficient simulation of the flow with non-asymptotic convergence rate.
Study Langevin Monte Carlo for sampling non-log-concave distributions.
problem Sampling from non-log-concave distributions, especially Gaussian mixtures.
method Discretizations of overdamped Langevin diffusions.
result Numerical simulations compare Langevin Monte Carlo algorithms' performance.
Adaptive algorithm improves convergence rate of Langevin dynamics.
problem Improving convergence rate of Langevin dynamics.
method Adaptive non-reversible stochastic gradient Langevin dynamics algorithm.
result Improved convergence rate of the algorithm.
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference. In this work we perform an analytic study of the performances of one of them, the Langevin algorithm, in the context of noisy high-dimensional inference. We employ the Langevin alg…
Study on Langevin dynamics convergence rates and their application to GAN training.
problem Understanding the long-term behavior of Langevin dynamics equations.
method Analytical and numerical methods to study convergence rates of underdamped mean-field Langevin dynamics.
result Exponential convergence rate results for the Langevin dynamics under various conditions.
Improved Langevin algorithms with prior diffusion achieve dimension-independent convergence for non-log-concave distributions.
problem Understanding the dimension dependency of computational complexity in high-dimensional sampling.
method Investigation of prior diffusion technique for log-Sobolev inequality target distributions.
result Modified Langevin algorithm achieves dimension-independent KL divergence convergence.
Adaptive Langevin dynamics reduces bias in Bayesian inference with mini-batching.
problem Bias in posterior sampling due to mini-batching in Bayesian inference.
method Adaptive Langevin dynamics with dynamical friction to correct noise.
result Quantified bias in posterior distribution due to mini-batching.
Langevin autoencoders improve deep latent variable models with efficient posterior sampling.
problem Efficient posterior sampling in deep latent variable models using MCMC.
method Amortized Langevin dynamics (ALD) replaces datapoint-wise sampling with encoder updates.
result ALD is valid as an MCMC algorithm with the target posterior as a stationary distribution.
Langevin dynamics fails to produce accurate samples even with small score function errors.
problem Robustness of Langevin dynamics to score function errors.
method Analysis of Langevin dynamics and score function errors.
result Langevin dynamics produces a distribution far from the target distribution in TV distance even with small L 2 L^2 L 2 errors in the score function. Study on convergence of Langevin dynamics for zero-sum games in probability distributions.
problem Analyzing convergence of Langevin dynamics for zero-sum games in probability distributions.
method Proved exponential and biased convergence guarantees for mean-field and finite-particle min-max Langevin dynamics.
result Explicit iteration complexity for finite-particle algorithms to approximate equilibrium distributions.
Framework infers Langevin dynamics from stochastic observations of latent systems.
problem Inferring non-stationary Langevin dynamics from indirect stochastic observations.
method Non-parametric framework explicitly modeling stochastic observation process and non-stationary latent dynamics.
result Correct inference of non-stationary dynamics requires accounting for non-equilibrium states and observation duration.
Recent works have derived non-asymptotic upper bounds for convergence of underdamped Langevin MCMC. We revisit these bound and consider introducing scaling terms in the underlying underdamped Langevin equation. In particular, we provide conditions under which an appropriate scaling allows to improve the error bounds in…
Estimates generalization gap for overparameterized models using Langevin approximation.
problem Estimating the difference between training and generalization performance in overparameterized models.
method Functional variance and Langevin approximation of functional variance.
result Demonstrates efficient estimation of generalization gaps for overparameterized models.
A new algorithm solves semidefinite programs using Langevin diffusion.
problem Optimizing semidefinite programs with diagonal constraints.
method Langevin diffusion on a product manifold of spheres.
result Langevin algorithm achieves ε accuracy in Ω(ε^-5) iterations.
Improved KLMC for sampling under various conditions.
problem Stable simulation of kinetic Langevin dynamics under different parameters.
method Revisited synchronous Wasserstein coupling analysis with stochastic exponential Euler discretization.
result Exponential integrator can simulate kinetic Langevin dynamics in the overdamped regime with proper time acceleration.
Study of diffusion annealed Langevin dynamics for generative models.
problem Theoretical efficiency of score-based diffusion processes.
method Rigorous construction and analysis of diffusion processes with Poincaré and logarithmic Sobolev inequalities.
result Improvement in efficiency of diffusion processes through Poincaré and logarithmic Sobolev inequalities.
DALMC provides non-asymptotic error bounds for generative models.
problem Efficiently generating samples from complex data distributions.
method Analysis of diffusion paths and Langevin Monte Carlo.
result Theoretical guarantees for a class of generative models.
Paper analyzes and compares ELF algorithms for federated learning.
problem Improving efficiency and privacy in federated learning.
method Proposes P-ELF, D-ELF, and B-ELF algorithms with primal, dual, and bidirectional compression.
result Provides non-asymptotic convergence guarantees under Log-Sobolev inequality.
Noise-corrected Langevin algorithm improves sampling from noisy data.
problem Sampling from noisy data with biased score function.
method Noise-corrected Langevin algorithm using noisy score function.
result Bias due to noisy data is removed, improving sampling accuracy.
New Langevin method achieves third order convergence for strongly log-concave distributions.
problem Sampling from complex distributions efficiently.
method Underdamped Langevin diffusion with third order convergence.
result Achieves 2-Wasserstein error of ε in O(√d/ε^1/3) steps under additional Lipschitz condition.