Unified view on learning unnormalized distributions using NCE.
problem Learning unnormalized distributions across different communities.
method Noise-Contrastive Estimation (NCE) for a unified perspective.
result Established finite-sample convergence rates for exponential families.
REGS samples from unnormalized distributions using gradient flow and neural networks.
problem Sampling from unnormalized distributions with high accuracy and efficiency.
method REGS is a particle method that iteratively transforms samples from a reference distribution to match an unnormalized target distribution using Wasserstein gradient flow and neural networks.
result REGS outperforms state-of-the-art methods in sampling from challenging multimodal distributions and real datasets.
Several statistical models are given in the form of unnormalized densities, and calculation of the normalization constant is intractable. We propose estimation methods for such unnormalized models with missing data. The key concept is to combine imputation techniques with estimators for unnormalized models including no…
Paper optimizes change detection in unnormalized distributions.
problem Detecting changes in unnormalized pre- and post-change distributions.
method Log-Partition Approximation Cumulative Sum (LPA-CUSUM) algorithm based on thermodynamic integration.
result Asymptotically optimal performance achieved through unbiased estimation of CUSUM statistics.
We investigate adversarial learning in the case when only an unnormalized form of the density can be accessed, rather than samples. With insights so garnered, adversarial learning is extended to the case for which one has access to an unnormalized form u(x) of the target density function, but no samples. Further, new c…
CD algorithm achieves near-optimal convergence rate for unnormalized models.
problem Training unnormalized models with high efficiency.
method Non-asymptotic analysis of contrastive divergence algorithm.
result CD can achieve O(n−1/2) convergence rate under regularity assumptions. New CUSUM algorithm detects changes in unnormalized models.
problem Change detection in models with unknown normalization constants.
method Score-based CUSUM (SCUSUM) algorithm based on Fisher divergence and Hyvärinen score.
result Asymptotic optimality of the SCUSUM algorithm demonstrated.
PDDS samples from unnormalized densities using iterative particle scheme.
problem Sampling from unnormalized probability densities.
method Iterative particle scheme with novel score matching loss.
result Asymptotically consistent estimates for multimodal and high-dimensional tasks.
A new method called TemperFlow tackles multimodality in sampling from unnormalized distributions.
problem Sampling from unnormalized distributions with isolated modes.
method TemperFlow learns a sequence of tempered distributions to progressively approach the target distribution.
result TemperFlow overcomes the limitations of existing methods and achieves superior performance.
Unified empirical and variational Bayes for unnormalized densities.
problem Approximating unnormalized densities using latent variable models.
method Formulate a latent variable model for Y=X+N(0,σ2Id), use ELBO as parametrization of Y's energy function, and estimate X with empirical Bayes least-squares. result UVB has higher capacity to approximate energy functions than MLPs in DEEN.
Single-step samplers generate high-quality samples efficiently.
problem Sampling from unnormalized distributions is computationally expensive.
method Developed consistent diffusion samplers that generate samples in a single step.
result Single-step samplers produce high-fidelity samples with less than 1% of traditional samplers' evaluations.
We develop underdamped diffusion bridges for sampling from unnormalized densities.
problem Sampling from unnormalized densities without direct access to samples.
method Underdamped diffusion bridges with rigorous score matching equivalence.
result State-of-the-art performance in sampling across various problems.
Proposes a method combining CNFs and rejection-resampling for sampling from unnormalized densities.
problem Sampling from unnormalized probability densities, especially multimodal ones.
method Combines continuous normalizing flows with rejection-resampling steps based on importance weights.
result The method improves sampling accuracy and performance compared to state-of-the-art methods.
SoftCVI uses contrastive estimation to infer complex posteriors.
problem Estimating complex posteriors in Bayesian inference.
method Contrastive variational inference with self-generated soft labels.
result SoftCVI outperforms other variational approaches in stability and coverage.
LFIS uses a time-dependent velocity field to sample from complex distributions.
problem Sampling from unnormalized density functions.
method LFIS learns a time-dependent velocity field to transport samples from a simple initial distribution to a complex target distribution.
result LFIS achieves state-of-the-art performance on various benchmark problems.
A new sequential test for unnormalized densities.
problem Testing unnormalized densities with adaptive stopping.
method Sequential kernelized Stein discrepancy test, using non-uniform Stein kernels.
result Valid test with asymptotic lower bound for growth.
CR-AIS improves AIS efficiency by constant rate annealing.
problem Efficiently sample from intractable distributions.
method Constant rate annealing schedule for AIS.
result CR-AIS outperforms existing Adaptive AIS methods.
MT-SGD samples from multiple target distributions using gradient descent.
problem Sampling from multiple unnormalized target distributions.
method Proposes MT-SGD, a flow of intermediate distributions to sample from multiple target distributions.
result Asymptotic analysis shows MT-SGD reduces to multiple-gradient descent for multi-objective optimization.
A new method for sampling from complex distributions using Langevin samplers.
problem Sampling from unnormalized Boltzmann densities.
method Probability flow ODE derived from linear stochastic interpolants, employing Langevin samplers.
result Efficient simulation of the flow with non-asymptotic convergence rate.
A new method optimizes a generalized Kullback-Leibler divergence for better simulation-based inference.
problem Optimizing likelihood functions when they are only known implicitly.
method Optimizes a generalized Kullback-Leibler divergence that accounts for normalization constants in unnormalized distributions.
result Unified approach that combines Neural Posterior Estimation and Neural Ratio Estimation.
iEFM trains CNF models from unnormalized densities efficiently.
problem Training generators from energy functions or unnormalized densities.
method Iterated energy-based flow matching (iEFM) with simulation-free objective.
result iEFM outperforms existing methods in probabilistic modeling.
New method for sampling from complex distributions using stochastic localization.
problem Sampling from unnormalized target densities in multi-modal distributions.
method Stochastic Localization via Iterative Posterior Sampling (SLIPS) framework.
result Approximate samples from target distribution and denoiser learned iteratively.
Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using …
The parameter estimation of unnormalized models is a challenging problem. The maximum likelihood estimation (MLE) is computationally infeasible for these models since normalizing constants are not explicitly calculated. Although some consistent estimators have been proposed earlier, the problem of statistical efficienc…
A new variational inference method using sliced Wasserstein distance is proposed.
problem The inefficiency and unreasonable properties of Kullback-Leibler divergence.
method Minimizing sliced Wasserstein distance, a valid metric from optimal transport.
result The proposed method approximates the unnormalized distribution efficiently and without requiring a tractable density function.
There are many models, often called unnormalized models, whose normalizing constants are not calculated in closed form. Maximum likelihood estimation is not directly applicable to unnormalized models. Score matching, contrastive divergence method, pseudo-likelihood, Monte Carlo maximum likelihood, and noise contrastive…
Morse neural networks improve uncertainty quantification and detection.
problem Uncertainty quantification and out-of-distribution detection.
method Generalizes unnormalized Gaussian densities to high-dimensional submanifolds using KL-divergence loss.
result Unified approach for OOD detection, anomaly detection, and continuous learning.
We present a new method for evaluating and training unnormalized density models. Our approach only requires access to the gradient of the unnormalized model's log-density. We estimate the Stein discrepancy between the data density p(x) and the model density q(x) defined by a vector function of the data. We paramete…
New methods improve memory efficiency for sampling from complex distributions.
problem Sampling from complex unnormalized distributions over discrete domains.
method Two novel training methods for discrete diffusion samplers.
result Achieve state-of-the-art results in unsupervised combinatorial optimization.
Method estimates Bayesian evidence from posterior samples using normalizing flows.
problem Estimating Bayesian evidence from posterior samples.
method Normalizing flows for evidence estimation.
result Method is more robust to sharp features in posterior distributions, especially in higher dimensions.
DDS samples from noisy data by reversing diffusion, providing theoretical guarantees.
problem Sampling from unnormalized densities.
method Denoising diffusion process, score matching, optimal control, Schrödinger bridges.
result DDS provides theoretical guarantees for sampling.
We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and sco…
Improved diffusion bridge sampling with rKL-LD loss.
problem Improving sampling from unnormalized distributions using diffusion bridges.
method Employing the rKL-LD loss instead of the Log Variance (LV) loss for diffusion bridges.
result rKL-LD consistently outperforms LV loss in diffusion bridges.
New method minimizes robust density power-based divergences for general parametric densities.
problem Computational complexity of minimizing DPD for general parametric densities.
method Stochastic approach to minimize DPD for general parametric density models.
result Proposed method can be applied to minimize other density power-based γ-divergences.
Unified framework for data-free sampling using Wasserstein gradient flows.
problem Efficient sampling from unnormalized distributions without data.
method Unified theoretical framework based on Wasserstein gradient flows.
result Unified form of velocity field for various f-divergences.
Score matching provides an effective approach to learning flexible unnormalized models, but its scalability is limited by the need to evaluate a second-order derivative. In this paper, we present a scalable approximation to a general family of learning objectives including score matching, by observing a new connection …
We introduce a new family of estimators for unnormalized statistical models. Our family of estimators is parameterized by two nonlinear functions and uses a single sample from an auxiliary distribution, generalizing Maximum Likelihood Monte Carlo estimation of Geyer and Thompson (1992). The family is such that we can e…
Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples from the other. Stein discrepancies can be combined with kernels to define kernelized Stein discrepanc…
A problem about Khovanov homology and 3-manifolds is discussed in this paper.
New sampling method uses gradient-free IPS with RKHS velocity field.
problem Efficient sampling from unnormalized target densities.
method Gradient-free interacting particle systems (IPS) with RKHS velocity field.
result IPS produce high-quality samples from various target distributions.
New complexity analysis for estimating normalizing constants in high dimensions.
problem Estimating the normalizing constant of unnormalized probability densities in high dimensions.
method Analyze and derive the oracle complexity of annealed importance sampling.
result Oracle complexity of $\widetilde{O}\left(\frac{dβ^2{\mathcal{A}}^2}{\varepsilon^4}
ight)$ for estimating Z within ε relative error. Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al., CoNLL 2019] for exploiting global properties of sequences for data-efficient learning of seq2seq models. In the first phase of training, an Energy-Based model (EBM) over sequences is derived. This EBM has high representational power, but is u…
We propose a simple algorithm to train stochastic neural networks to draw samples from given target distributions for probabilistic inference. Our method is based on iteratively adjusting the neural network parameters so that the output changes along a Stein variational gradient direction (Liu & Wang, 2016) that maxima…
New sampling method uses stochastic interpolants and FBSDEs.
problem Sampling from high-dimensional distributions with unnormalized densities.
method Stochastic interpolants and FBSDEs to define and solve diffusion process.
result Effective sampling from challenging distributions.
A new method for reconstructing flows from perturbed distributions.
problem Reconstructing flows from perturbed probability distributions.
method Integrable vector fields and Green's functions.
result A nonparametric flow can be computed to generate samples from a perturbed distribution.
New method trains neural samplers to sample from multi-modal distributions efficiently.
problem Mode-seeking behavior of reverse KL divergence hinders effective sampling from multi-modal target distributions.
method Minimizing reverse diffusive KL divergence along diffusion trajectories of model and target densities.
result Demonstrated enhanced sampling performance across various multi-modal distributions.
Stable GFlowNets prevent loss spikes and mode collapse in training.
problem Unstable training of GFlowNets leading to loss spikes and mode collapse.
method Assessed sensitivity of GFlowNet objectives, derived loss-to-TV bounds, and proposed Stable GFlowNets.
result Stable GFlowNets improve training behavior and distributional fidelity.
Unified framework for OOD detection using class ratio estimation.
problem Density-based OOD detection is unreliable for OOD images.
method Unified framework that builds energy-based models and employs differing base distributions, directly estimating the density ratio through class ratio estimation.
result Competitive results on OOD image problems compared to recent work.