arXiv research

The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propag…

In this paper, we face the problem of simulating discrete random variables with general and varying distributions in a scalable framework, where fully parallelizable operations should be preferred. The new paradigm is inspired by the context of discrete choice models. Compared to classical algorithms, we add paralleliz…

Simplified analysis of diffusion models using discrete random variables.

Estimating the gradients of stochastic nodes in stochastic computational graphs is one of the crucial research questions in the deep generative modeling community, which enables the gradient descent optimization on neural network parameters. Stochastic gradient estimators of discrete random variables are widely explore…

Efficient algorithm approximates discrete random variables with minimal Kolmogorov distance.

Probabilistic programs with mixed support (both continuous and discrete latent random variables) commonly appear in many probabilistic programming systems (PPSs). However, the existence of the discrete random variables prohibits many basic gradient-based inference engines, which makes the inference procedure on such mo…

We introduce Network Maximal Correlation (NMC) as a multivariate measure of nonlinear association among random variables. NMC is defined via an optimization that infers transformations of variables by maximizing aggregate inner products between transformed variables. For finite discrete and jointly Gaussian random vari…

Derives derivatives of risk measures for various types of portfolio losses.

New criteria distinguish cause from effect in data, overcoming statistical limitations.

Discrete random variables are natural components of probabilistic clustering models. A number of VAE variants with discrete latent variables have been developed. Training such methods requires marginalizing over the discrete latent variables, causing training time complexity to be linear in the number clusters. By appl…

Reparameterization of variational auto-encoders with continuous random variables is an effective method for reducing the variance of their gradient estimates. In the discrete case, one can perform reparametrization using the Gumbel-Max trick, but the resulting objective relies on an $\arg \max$ operation and is non-dif…

Information bottleneck (IB) is a technique for extracting information in one random variable $X$ that is relevant for predicting another random variable $Y$. IB works by encoding $X$ in a compressed "bottleneck" random variable $M$ from which $Y$ can be accurately decoded. However, finding the optimal bottleneck variab…

New tree-structured Markov fields with Poisson marginals for counting variables.

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discret…

Bayesian optimization adapted for discrete spaces using random mappings.

IDF++ improves integer discrete flows for lossless compression.

Low-variance gradient estimation is crucial for learning directed graphical models parameterized by neural networks, where the reparameterization trick is widely used for those with continuous variables. While this technique gives low-variance gradient estimates, it has not been directly applicable to discrete variable…

New gradient estimators for discrete variables improve model training.

Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.

Foster and Hart proposed an operational measure of riskiness for discrete random variables. We show that their defining equation has no solution for many common continuous distributions including many uniform distributions, e.g. We show how to extend consistently the definition of riskiness to continuous random variabl…

We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators. Combining our estimator with REINFORCE, we ob…

A new graphical model for discrete data without parametric restrictions.

In this study, we propose a new definition of multivariate conditional value-at-risk (MCVaR) as a set of vectors for discrete probability spaces. We explore the properties of the vector-valued MCVaR (VMCVaR) and show the advantages of VMCVaR over the existing definitions given for continuous random variables when adapt…

Estimates treatment effects in randomized experiments with non-compliance.

Sequence-to-sequence text-to-speech (TTS) is dominated by soft-attention-based methods. Recently, hard-attention-based methods have been proposed to prevent fatal alignment errors, but their sampling method of discrete alignment is poorly investigated. This research investigates various combinations of sampling methods…

We adapt a Markov Random Field learning algorithm for continuous variables.

Optimized variable orderings improve autoregressive model performance.

GCNs converge and remain stable on large random graphs, revealing geometric insights.

In this manuscript, we analytically and numerically study statistical properties of an heteroskedastic process based on the celebrated ARCH generator of random variables whose variance is defined by a memory of $q_{m}$-exponencial, form ($e_{q_{m}=1}^{x}=e^{x}$). Specifically, we inspect the self-correlation function o…

We provide a conceptual map to navigate causal analysis problems. Focusing on the case of discrete random variables, we consider the case of causal effect estimation from observational data. The presented approaches apply also to continuous variables, but the issue of estimation becomes more complex. We then introduce …

Generative model for joint discrete distributions using randomized assignment flows.

Criterion extends identifiability for continuous mixtures of kernels.

Lognormal random variables appear naturally in many engineering disciplines, including wireless communications, reliability theory, and finance. So, too, does the sum of (correlated) lognormal random variables. Unfortunately, no closed form probability distribution exists for such a sum, and it requires approximation. …

We examine Kreps' (2019) conjecture that optimal expected utility in the classic Black--Scholes--Merton (BSM) economy is the limit of optimal expected utility for a sequence of discrete-time economies that "approach" the BSM economy in a natural sense: The $n$th discrete-time economy is generated by a scaled $n$-step r…

The Straight-Through (ST) estimator is a widely used technique for back-propagating gradients through discrete random variables. However, this effective method lacks theoretical justification. In this paper, we show that ST can be interpreted as the simulation of the projected Wasserstein gradient flow (pWGF). Based on…

We study the problem of identifying the causal relationship between two discrete random variables from observational data. We recently proposed a novel framework called entropic causality that works in a very general functional model but makes the assumption that the unobserved exogenous variable has small entropy in t…

This paper provides a tutorial on Boltzmann Machines and Deep Belief Networks.

CARMS improves gradient estimation for categorical variables.

Proposes ABC method for discrete data, improving likelihood-free inference.

Inferring the causal structure of a set of random variables from a finite sample of the joint distribution is an important problem in science. Recently, methods using additive noise models have been suggested to approach the case of continuous variables. In many situations, however, the variables of interest are discre…

Optimal transport is #P-hard when components are independent, even with approximate solutions.

CLOUD method detects causal relationships in various data types without latent variable assumptions.

We introduce a parameterization method called Neural Bayes which allows computing statistical quantities that are in general difficult to compute and opens avenues for formulating new objectives for unsupervised representation learning. Specifically, given an observed random variable $\mathbf{x}$ and a latent discrete …

CIPNN model tackles continuous latent variables, solving intractable posterior problems.

The paper confirms a conjecture about optimal expected utility in markets with insider information.

Drawing a sample from a discrete distribution is one of the building components for Monte Carlo methods. Like other sampling algorithms, discrete sampling suffers from the high computational burden in large-scale inference problems. We study the problem of sampling a discrete random variable with a high degree of depen…

Recent developments have linked causal inference with Algorithmic Information Theory, and methods have been developed that utilize Conditional Kolmogorov Complexity to determine causation between two random variables. We present a method for inferring causal direction between continuous variables by using an MDL Binnin…

This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness. The PRGDS is based on a new motif in Bayesian latent variable modeling, an alternating chain of discrete Poisson and continuous …