We develop underdamped diffusion bridges for sampling from unnormalized densities.
problem Sampling from unnormalized densities without direct access to samples.
method Underdamped diffusion bridges with rigorous score matching equivalence.
result State-of-the-art performance in sampling across various problems.
Recent works have derived non-asymptotic upper bounds for convergence of underdamped Langevin MCMC. We revisit these bound and consider introducing scaling terms in the underlying underdamped Langevin equation. In particular, we provide conditions under which an appropriate scaling allows to improve the error bounds in…
Error estimates found between SGD with momentum and Langevin diffusion.
problem Quantifying the difference between SGD with momentum and Langevin diffusion.
method Established error estimates using 1-Wasserstein and total variation distances.
result Quantitative error estimates between SGD with momentum and underdamped Langevin diffusion.
New method optimizes non-linear functionals over probability measures.
problem Optimizing non-linear functionals defined over probability measures.
method N-particle underdamped Langevin algorithm with spacetime discretization.
result Converges globally in total variation distance.
Study on Langevin dynamics convergence rates and their application to GAN training.
problem Understanding the long-term behavior of Langevin dynamics equations.
method Analytical and numerical methods to study convergence rates of underdamped mean-field Langevin dynamics.
result Exponential convergence rate results for the Langevin dynamics under various conditions.
New method controls bias in unadjusted Hamiltonian Monte Carlo and underdamped Langevin.
problem Bias in unadjusted Hamiltonian Monte Carlo and underdamped Langevin samplers.
method Delocalization of bias technique applied to these samplers.
result Control W 2 W_2 W 2 bias with O ( K ) O(\sqrt{K}) O ( K ) integration steps for high-dimensional distributions. We study the underdamped Langevin diffusion when the log of the target distribution is smooth and strongly concave. We present a MCMC algorithm based on its discretization and show that it achieves ε \varepsilon ε error (in 2-Wasserstein distance) in O ( d / ε ) \mathcal{O}(\sqrt{d}/\varepsilon) O ( d / ε ) steps. This is a significant improv…
New method proves dimension-free convergence for ULD in KL divergence.
problem Polynomial scaling of existing convergence guarantees in high dimensions.
method Refined KL local error framework, focusing on tr(H) instead of d.
result First dimension-free KL divergence bounds for discretized ULD.
Improved sampling guarantees for underdamped Langevin Monte Carlo without restrictive assumptions.
problem Sampling from unnormalized densities with improved guarantees and acceleration.
method Novel analysis relaxing assumptions on log-Sobolev inequality and Hessian smoothness, using Rényi discretization bounds.
result First KL divergence guarantees for ULMC without Hessian smoothness under strong log-concavity.
Improved sampling for high-dimensional posteriors with underdamped Langevin.
problem Scalability issues in high-dimensional problems with approximate Thompson sampling.
method Underdamped Langevin Monte Carlo for accelerated posterior concentration.
result Logarithmic regret improvement from i l d e O ( d ) \mathcal{ ilde O}(d) i l d e O ( d ) to i l d e O ( d ) \mathcal{ ilde O}(\sqrt{d}) i l d e O ( d ) . New Langevin method achieves third order convergence for strongly log-concave distributions.
problem Sampling from complex distributions efficiently.
method Underdamped Langevin diffusion with third order convergence.
result Achieves 2-Wasserstein error of ε in O(√d/ε^1/3) steps under additional Lipschitz condition.
A new sampling method reduces computational cost for high-dimensional log-concave distributions.
problem High computational cost of ULMC in high dimensions.
method Random Coordinate ULMC (RC-ULMC) selects a single coordinate per iteration.
result RC-ULMC is cheaper than classical ULMC, especially in highly skewed and high-dimensional problems.
Develops unbiased estimation method using underdamped Langevin dynamics.
problem Estimating expectations of non-negative Lebesgue density probability measures.
method Underdamped Langevin dynamics, time-discretized versions, doubly randomized estimation.
result Proves finite variance and expected/finite cost of the proposed estimator.
We formulate gradient-based Markov chain Monte Carlo (MCMC) sampling as optimization on the space of probability measures, with Kullback-Leibler (KL) divergence as the objective functional. We show that an underdamped form of the Langevin algorithm performs accelerated gradient descent in this metric. To characterize t…
Unified framework for constrained diffusion models on nonconvex sets with efficient landing mechanism.
problem Efficiently modeling generative models under nonconvex constraints.
method Unified framework with overdamped and underdamped dynamics, landing mechanism.
result Significantly reduces computational cost while maintaining sample quality.
New algorithms improve sampling from constrained distributions.
problem Sampling from distributions constrained to convex bodies.
method Penalized Langevin Dynamics and Underdamped Monte Carlo methods.
result Improved convergence rates for constrained sampling problems.
Enhances LMC for log-concave sampling, reducing computational cost.
problem High computational cost of LMC for high-dimensional problems.
method Random coordinate descent (RCD) combined with variance reduction techniques (SAGA, SVRG).
result Achieves computational cost reduction compared to classical LMC, same number of iterations as LMC.
Discretizations of Langevin diffusions provide a powerful method for sampling and Bayesian inference. However, such discretizations require evaluation of the gradient of the potential function. In several real-world scenarios, obtaining gradient evaluations might either be computationally expensive, or simply impossibl…
Study on Wasserstein distance for numerical approximations of stochastic differential equations.
problem Estimating the Wasserstein distance between stochastic differential equation distributions and their numerical approximations.
method Unified framework for analyzing different integrators and a novel splitting method for underdamped Langevin dynamics.
result A novel splitting method for underdamped Langevin dynamics with optimal complexity.
KIPLMC methods improve statistical inference in latent variable models.
problem Statistical inference in latent variable models.
method Joint diffusion process in parameter and latent variable spaces, with two explicit discretizations.
result KIPLMC methods achieve accelerated convergence rates in Wasserstein-2 distance.
We study the problem of sampling from a distribution p ∗ ( x ) ∝ exp ( − U ( x ) ) p^*(x) \propto \exp\left(-U(x)\right) p ∗ ( x ) ∝ exp ( − U ( x ) ) , where the function U U U is L L L -smooth everywhere and m m m -strongly convex outside a ball of radius R R R , but potentially nonconvex inside this ball. We study both overdamped and underdamped Langevin MCMC and establish upper bound…
New method identifies physical constants from video data alone.
problem Identifying physical constants from video data.
method Proves level-set slope-coverage condition ensures local affine mapping to true physical state, enabling exact parameter recovery.
result Underdamped systems identifiable from a single video clip, other regimes require three diverse trajectories.
The paper analyzes convergence of Langevin dynamics with time-dependent metrics.
problem Analyzing convergence of Langevin dynamics with time-dependent metrics.
method Formulated a modified gradient flow of the Kullback-Leibler divergence, selected a time-dependent relative Fisher information functional, and developed a time-dependent Hessian matrix condition.
result Proved convergence conditions for various Langevin dynamics.
Improved KLMC for sampling under various conditions.
problem Stable simulation of kinetic Langevin dynamics under different parameters.
method Revisited synchronous Wasserstein coupling analysis with stochastic exponential Euler discretization.
result Exponential integrator can simulate kinetic Langevin dynamics in the overdamped regime with proper time acceleration.
Enhanced Gaussian process regression for multi-fidelity data fusion.
problem Combining data of varying fidelity levels for accurate predictions.
method Gradient-enhanced Cokriging method (GE-Cokriging) for QoI and its gradients.
result GE-Cokriging outperforms conventional multi-fidelity Cokriging in predicting QoI and gradients.
Stochastic gradient descent with momentum (SGDm) is one of the most popular optimization algorithms in deep learning. While there is a rich theory of SGDm for convex problems, the theory is considerably less developed in the context of deep learning where the problem is non-convex and the gradient noise might exhibit a…
New method reduces variance in random coordinate descent for Langevin Monte Carlo.
problem Efficient sampling from log-concave distributions in high dimensions.
method Introduces RCAD, a variance reduction technique for RCD-LMC.
result RCAD-O-LMC and RCAD-U-LMC converge within the same number of iterations as classical LMC methods, saving computational cost.
Anomalous diffusion in SGD reveals interactions between hyperparameters and Hessian.
problem Understanding the limiting dynamics of SGD in deep neural networks.
method Continuous-time model of SGD as an underdamped Langevin equation, derived for linear regression.
result Anomalous diffusion is explained by modified loss and probability currents in phase space.
Generative models learn smoother densities to sample from unknown distributions.
problem Sampling from unknown distributions in high-dimensional spaces.
method Formalizes sampling problem, introduces multimeasurement noise model, derives Bayes estimator, and uses underdamped Langevin MCMC.
result Formulation leads to efficient sampling methods and theoretical connections with denoising autoencoders.
A new framework RTK accelerates diffusion inference by breaking down the process into fewer, more efficient subproblems.
problem Efficiently generating data from trained diffusion models using discretized reverse SDEs or ODEs.
method Developed a general RTK framework that decomposes the diffusion process into fewer, more balanced subproblems, using MALA and ULD for sampling.
result The RTK-MALA and RTK-ULD algorithms achieve faster convergence rates and lower error compared to existing methods.
We provide convergence guarantees in Wasserstein distance for a variety of variance-reduction methods: SAGA Langevin diffusion, SVRG Langevin diffusion and control-variate underdamped Langevin diffusion. We analyze these methods under a uniform set of assumptions on the log-posterior distribution, assuming it to be smo…
Paper proposes new Langevin samplers for sampling from log-concave distributions with superlinear gradient growth.
problem Sampling from log-concave distributions with superlinear gradient growth.
method Proposes two novel discretizations of kinetic Langevin SDEs, showing contractivity and log-Sobolev inequality.
result Establishes non-asymptotic bounds in 2-Wasserstein distance between sampled distributions and target measures.
Study on convergence of SDEs using entropy methods.
problem Analyzing convergence of stochastic differential equations.
method Applied Lyapunov method to Fokker-Planck equation with weighted relative Fisher information.
result Exponential convergence of probability density function to invariant distribution in L 1 L_1 L 1 distance. Improved log-concave sampling to O ( d 1 / 2 ) O(d^{1/2}) O ( d 1/2 ) with warm starts.
problem Sampling from strongly log-concave distributions efficiently.
method Warm starts and discretized underdamped Langevin diffusion.
result Achieved O ( d 1 / 2 ) O(d^{1/2}) O ( d 1/2 ) complexity for high-accuracy sampling. This paper shows how to approximate any log-concave distribution using well-conditioned affine coupling flows.
problem Understanding the representational power of affine coupling flows for log-concave distributions.
method Leveraging connections between affine coupling architectures, Langevin dynamics, and Hénon maps to prove log-concave approximation.
result Any log-concave distribution can be approximated using well-conditioned affine-coupling flows.
New method speeds up generative modeling without requiring diffusion steps.
problem Improving the speed and efficiency of generative modeling techniques.
method Probability flow ODE with a corrector step, achieving better dimension dependence.
result Better dimension dependence ( O ( d ) O(\sqrt{d}) O ( d ) vs. O ( d ) O(d) O ( d ) , assuming smoothness of the data distribution). Proposes methods to add constraints to neural networks to improve stability and generalization.
problem Improving stability and generalization of neural networks.
method Constraint-based regularization using stochastic gradient Langevin dynamics.
result Constraints help stabilize and improve the robustness of deep neural networks.
Efficient event generation for collider phenomenology using parallel Langevin sampling and learned Stein diagnostics.
problem Event generation for precision collider phenomenology.
method Parallel Langevin sampling with learned Stein diagnostics.
result Relaxation time is estimated using a data-driven approach.
Estimating the normalizing constant of an unnormalized probability distribution has important applications in computer science, statistical physics, machine learning, and statistics. In this work, we consider the problem of estimating the normalizing constant Z = ∫ R d e − f ( x ) d x Z=\int_{\mathbb{R}^d} e^{-f(x)}\,\mathrm{d}x Z = ∫ R d e − f ( x ) d x to within a m…
Proposes a new method for constrained generative modeling using Langevin dynamics.
problem Challenges in satisfying underlying constraints with score-based generative models.
method Uses kinetic Langevin dynamics with specular reflection to model constraints.
result Demonstrates efficient numerical samplers with optimal convergence rates.
The paper analyzes the randomized midpoint method for Langevin diffusions, revealing biases and asymptotic properties.
problem Analyzing biases and asymptotic properties of the randomized midpoint method for Langevin diffusions.
method Characterization of stationary distribution and asymptotic normality for numerical integration.
result The step-size needs to go to zero for the method to be asymptotically unbiased.
Constraints improve deep neural network training by stabilizing and enhancing robustness.
problem Vanishing/exploding gradients and poor weight magnitudes in deep neural networks.
method Weight-constrained stochastic dynamics using Langevin dynamics framework.
result Enhanced exploration of the loss landscape and improved generalization.
This research accelerates sampling methods using Nesterov's Acceleration.
problem Improving sampling efficiency in MCMC methods.
method Developed a Hessian-Free High-Resolution ODE reformulation of NAG-SC, injected noise, and discretized the diffusion process.
result Quantified acceleration beyond underdamped Langevin in W 2 W_2 W 2 distance for log-strongly-concave targets. Parallel sampling for smooth distributions with fast convergence.
problem Efficiently sampling from distributions with smooth densities.
method Parallelization of Langevin algorithms under log-Sobolev inequalities.
result Samples close to target distribution with low KL divergence or TV distance.
A new RG approach connects discrete and continuous time descriptions of Gaussian processes.
problem Discretization of continuous stochastic processes for accurate simulation or model inference.
method Renormalization Group (RG) approach for Gaussian time series generated by auto-regressive models.
result RG fixed points correspond to discretizations of linear SDEs, providing insights into process accuracy.
A new sampling method estimates scores without training or nested MCMC.
problem Efficient sampling from complex, unnormalised distributions.
method Multiscale averaging in SDEs for score estimation.
result Empirical results show competitive accuracy and efficiency.
The paper explores efficient sampling for Bayesian wide neural networks.
problem Sampling from posterior distributions of wide neural networks.
method Preconditioned Crank-Nicolson and Langevin algorithms for reparametrised posterior distributions.
result The preconditioned Crank-Nicolson algorithm improves sampling efficiency in wide networks.
New modifiers improve noisy RNN replay in hippocampal networks.
problem Improving noisy RNN replay in hippocampal networks.
method Three approaches: hidden state leakage, adaptation, and momentum.
result Hidden state leakage, adaptation, and momentum improve noisy RNN replay.