Paper proposes a new generative model for discrete distributions using flows on submanifolds.
problem Discretization issues and complex statistical dependencies in discrete data.
method Continuous normalizing flows on factorizing discrete measures, geodesic flow matching.
result Efficient training and broad applicability demonstrated through experiments.
Paper develops a gradient-like proposal for discrete distributions without requiring natural differentiability.
problem Lack of natural differentiability in proposal distributions for discrete distributions.
method Locally-balanced proposal combined with Newton's series expansion for efficient exploration.
result Method guarantees convergence rate and outperforms alternatives in various experiments.
In this expository paper we illustrate the generality of game theoretic probability protocols of Shafer and Vovk (2001) in finite-horizon discrete games. By restricting ourselves to finite-horizon discrete games, we can explicitly describe how discrete distributions with finite support and the discrete pricing formulas…
Enhances gradient-based discrete samplers with parallel tempering for multimodal distributions.
problem Local minima in high-dimensional, multimodal discrete distributions.
method Combines parallel tempering with discrete Langevin proposal, using Metropolis criterion for swaps.
result Significantly faster mixing and better sampling from complex distributions.
We extend the Fourier cosine method to discrete probability distributions, achieving faster convergence rates.
problem Extending Fourier cosine method to discrete probability distributions.
method Spectral filters and convergence rates analysis.
result Spectral filters achieve one order faster convergence rates than previously recognized.
Develops a new method for learning discrete distributions without embedding them in a continuous space.
problem Challenges in learning discrete distributions using current methodologies.
method Introduces a MAD invertible map and a mixed variational flow (MAD Mix) for discrete distributions.
result MAD Mix produces more reliable approximations than continuous-embedding flows.
Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using…
Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD), provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD cannot be directly applied to discrete distributions. In this work, we fill this gap by propo…
SHIFT method optimally estimates heterogeneous discrete distributions with limited communication.
problem Collaborative learning of discrete distributions under heterogeneity and communication constraints.
method Two-stage method: First, users learn a central distribution; then, fine-tune this to estimate individual distributions.
result SHIFT is minimax optimal in the model of heterogeneity and under communication constraints.
Spectral clustering improves accuracy and efficiency for clustering discrete distributions.
problem Inaccurate clustering of discrete distributions using traditional methods.
method Spectral clustering combined with distribution affinity measures (MMD, Wasserstein distance) and linear optimal transport.
result Spectral clustering outperforms traditional methods in accuracy and efficiency.
Proposes an automatic cyclical scheduling for gradient-based discrete sampling.
problem Gradient-based sampling in high-dimensional models can get stuck in local modes.
method Cyclical step size and balancing schedules with automatic hyperparameter tuning.
result Proves non-asymptotic convergence and inference guarantees for general discrete distributions.
In this work we present Discrete Attend Infer Repeat (Discrete-AIR), a Recurrent Auto-Encoder with structured latent distributions containing discrete categorical distributions, continuous attribute distributions, and factorised spatial attention. While inspired by the original AIR model andretaining AIR model's capabi…
PDHAMS improves sampling for discrete distributions with quadratic potential functions.
problem Sampling discrete distributions efficiently and accurately.
method Integrates a second-order approximation of the potential function and uses Gaussian integral trick.
result PDHAMS yields superior performance compared to other methods.
While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-d…
Refined analysis of Mitra's algorithm for discrete mixtures.
problem Classifying general discrete mixture distribution models.
method Spectral clustering tailored to bipartite stochastic block models.
result Improved separation conditions for probability distributions.
EDLP samples flat modes in discrete spaces using entropy.
problem Sampling flat modes in discrete spaces is challenging.
method EDLP uses a continuous auxiliary variable and local entropy to guide sampling.
result EDLP consistently outperforms traditional methods in various tasks.
A new method for categorical variational inference using discrete normalizing flows.
problem Challenges in optimizing variational approximations for discrete latent variables.
method Differentiable reparameterization using a mixture of discrete normalizing flows.
result Improves optimization of evidence lower bound and reduces sensitivity to hyperparameters.
Unified discrete diffusion for categorical data simplifies training and sampling.
problem Training and sampling in discrete diffusion models for categorical data.
method Mathematical simplifications and elegant unification of discrete-time and continuous-time discrete diffusion.
result Unified Simplified Discrete Denoising Diffusion (USD3) outperforms SOTA baselines.
SDE automatically recovers interpretable discrete distributions.
problem Limited interpretable discrete probability laws.
method Unsupervised framework using symbolic density estimation.
result Accurately recovers interpretable discrete distributions.
AdaCat improves density estimation and planning in autoregressive models.
problem Efficiently modeling sharp density changes in continuous data.
method Adaptive Categorical Discretization (AdaCat) for autoregressive models.
result Improves density estimation and planning in various data types.
New formulae identify discrete probability laws without needing normalization constants.
problem Characterizing non-normalized discrete probability distributions.
method Derive explicit formulae for mass functions using Stein's method.
result Developed tools for solving statistical problems without normalization constants.
A new sampler for complex discrete distributions efficiently updates all variables in parallel.
problem Sampling complex high-dimensional discrete distributions efficiently and accurately.
method Discrete Langevin proposal (DLP) for parallel coordinate updates with controlled stepsize.
result DLP efficiently explores high-dimensional and strongly correlated variables with asymptotic bias of zero for log-quadratic distributions.
Generative model for joint discrete distributions using randomized assignment flows.
problem Efficiently representing and sampling from complex joint distributions of discrete variables.
method Randomized assignment flows on the statistical submanifold of factorizing distributions.
result Our model can efficiently represent and sample from any target distribution and assess likelihood of unseen data points.
Improves gradient estimation for discrete distributions with variance reduction techniques.
problem Excessive variance in gradient estimation for discrete distributions.
method Stein operators for discrete distributions and control variates.
result Substantially lower variance in gradient estimation.
Study error bounds in evaluating distributional computational graphs.
problem Error analysis in evaluating graphs with inputs as probability distributions.
method Establish non-asymptotic error bounds using Wasserstein-1 distance.
result Non-asymptotic error bounds for discretization errors in distributional computational graphs.
A new discrete formula connects vertex and edge distributions on graphs.
problem Optimal transport on graphs with mixed vertex and edge distributions.
method Discrete transport equation and Benamou-Brenier formulation.
result Classification of all Wasserstein-1 geodesics on graphs.
We present a mixed multinomial logit (MNL) model, which leverages the truncated stick-breaking process representation of the Dirichlet process as a flexible nonparametric mixing distribution. The proposed model is a Dirichlet process mixture model and accommodates discrete representations of heterogeneity, like a laten…
We present a framework for learning disentangled and interpretable jointly continuous and discrete representations in an unsupervised manner. By augmenting the continuous latent distribution of variational autoencoders with a relaxed discrete distribution and controlling the amount of information encoded in each latent…
Improved bounds for estimating discrete distributions in KL divergence.
problem Estimating discrete distributions in KL divergence with accuracy.
method Used Laplace estimator and established concentration bounds.
result Deviation from mean scales as k / n \sqrt{k}/n k / n for n ≥ k n \ge k n ≥ k . Optimal weights improve particle-based approximations of discrete distributions.
problem Improving particle-based approximations of discrete distributions.
method Proving optimality of weights and showing how to compute them efficiently.
result Optimal weights can be computed from existing particle-based methods without extra costs.
The study tackles modeling high-frequency financial data using continuous distributions, finding them inadequate.
problem Challenges in modeling high-frequency integer price changes with continuous distributions.
method Proposed a modified maximum likelihood estimation procedure to account for the discreteness of high-frequency price changes.
result Traditional GARCH models are not suitable for high-frequency data due to the discreteness of price changes.
HiSS sampling overcomes local mode traps in rugged discrete spaces.
problem Sampling multimodal discrete distributions with gradient-based methods.
method Integrates Metropolis-within-Gibbs framework with logistic convolution.
result HiSS outperforms alternatives on various tasks, including Ising models and binary neural networks.
Optimal SQ bounds for learning binary product distributions and Ising models.
problem Learning binary product distributions and Ising models robustly.
method Statistical Query (SQ) lower bounds for robust learning.
result Optimal SQ lower bounds match known algorithm error guarantees.
Paper solves NGCA for discrete distributions using LLL method.
problem Learning hidden non-Gaussian components in discrete distributions.
method Utilizes LLL lattice basis reduction method.
result Sample and computationally efficient algorithm for NGCA in discrete distributions.
New algorithm learns changing discrete distributions with minimal drift error.
problem Learning discrete distributions that change over time with limited past samples.
method Adaptive algorithm using data-dependent bounds to balance statistical and drift errors.
result Tighter statistical error bounds for drifting distributions with or without finite support.
Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for m…
Corrected samplers reduce discretization error in discrete flow models without additional computational cost.
problem Discretization error in samplers for discrete flow models.
method Established non-asymptotic error bounds for samplers, proposed time-corrected and location-corrected samplers.
result Location-corrected sampler has lower complexity and better generation quality.
Two EM algorithms estimate prior distributions in mixture of linear regressions.
problem Estimating prior distributions in mixture of linear regressions.
method Two EM algorithms: one for continuous priors, one for discrete priors.
result Both algorithms accurately estimate prior distributions and the number of clusters.
Unified error analysis for discrete flow models.
problem Error analysis of discrete flow models.
method Stochastic calculus theory, Girsanov theorem, generator matching, uniformization.
result First error analysis for discrete flow models.
FLDD improves discrete diffusion models by learning a non-Markovian noising process.
problem Efficiency and quality of discrete diffusion models in few-step generation.
method Introduces a learnable non-Markovian forward (noising) process to match the target distribution.
result FLDD produces higher quality samples in fewer steps compared to conventional discrete diffusion models.
DNFS trains efficient samplers for discrete distributions using locally equivariant Transformers.
problem Sampling from unnormalised discrete distributions.
method DNFS learns a rate matrix to satisfy the Kolmogorov equation, using control variates and locally equivariant Transformers.
result DNFS achieves efficient and effective sampling across various applications.
The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propag…
The paper analyzes the randomized midpoint method for Langevin diffusions, revealing biases and asymptotic properties.
problem Analyzing biases and asymptotic properties of the randomized midpoint method for Langevin diffusions.
method Characterization of stationary distribution and asymptotic normality for numerical integration.
result The step-size needs to go to zero for the method to be asymptotically unbiased.
We study proper losses for discrete generative models without knowing the target distribution.
problem Evaluating generative models in the discrete setting without direct access to the target distribution.
method Define and construct black-box proper losses using statistical estimation theory.
result Black-box proper losses must be of polynomial form and involve more samples than the polynomial degree.
Continuous time framework for discrete data denoising models.
problem Efficient training and sampling for discrete data denoising models.
method Formulated as Continuous Time Markov Chains (CTMCs), efficient training using continuous time ELBO, high-dimensional CTMC simulation, novel theoretical error bound.
result Continuous time treatment enables novel theoretical error bound between generated and true data distributions.
Sharp bounds for high-probability estimation of discrete distributions.
problem Estimating discrete distributions with high probability under χ 2 χ^2 χ 2 -divergence. method Sharp upper and lower bounds for the classical Laplace estimator, and characterization of minimax high-probability risk for any estimator.
result Sharp bounds for high-probability estimation of discrete distributions can be achieved through a simple smoothing strategy.
Discrete Gaussian noise preserves privacy and accuracy in differential privacy.
problem Finite computers cannot represent continuous Gaussian noise, leading to privacy breaches and loss of interpretability.
method Introduced and analyzed discrete Gaussian noise, providing privacy and accuracy guarantees similar to continuous Gaussian noise.
result Discrete Gaussian noise offers the same privacy and accuracy as continuous Gaussian noise, with efficient sampling algorithms.
Study shows how numerical discretization affects reconstructions and parameter distributions in nano metrology.
problem Impact of numerical discretization on parameter reconstructions and model parameter distributions.
method Bayesian target vector optimization, finite element model, Gaussian process, stochastic machine learning surrogate models, Markov chain Monte Carlo sampler.
result Numerical discretization parameters impact the accuracy and distribution of reconstructed model parameters.