arXiv research

End-to-end training of DBMs with improved gradient estimation.

Unbiased methods for alpha-divergence minimization struggle in high dimensions.

New algorithm finds unbiased subnetworks in biased datasets.

RényiCL uses Rényi divergence for robust contrastive learning with stronger data augmentations.

Improved UIVI method shows better performance than state-of-the-art SIVI methods.

MUSE provides unbiased stopping estimates for optimal problems.

Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training the weights of Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and…

Estimates proper calibration errors and refinement terms in probabilistic predictions.

SUMO provides unbiased log marginal likelihood estimation for latent variable models.

GANs excel at learning high dimensional distributions, but they can update generator parameters in directions that do not correspond to the steepest descent direction of the objective. Prominent examples of problematic update directions include those used in both Goodfellow's original GAN and the WGAN-GP. To formally d…

Paper resolves bias in ALFT training using generalized alignment games.

Theoretical proof shows COMs are a type of contrastive divergence model with improved sampling.

We consider the nonlinear Kalman filtering problem using Kullback-Leibler (KL) and $α$-divergence measures as optimization criteria. Unlike linear Kalman filters, nonlinear Kalman filters do not have closed form Gaussian posteriors because of a lack of conjugacy due to the nonlinearity in the likelihood. In this paper …

NCE and CD are shown to be equivalent ML methods.

Paper proposes f-EBM for training deep EBMs using various f-divergences.

This paper proposes a more efficient training method for energy-based models.

New method learns unbiased treatment representations from structured high-dimensional data.

CD algorithm achieves near-optimal convergence rate for unnormalized models.

New method improves model robustness to biased data.

We develop a method to combine Markov chain Monte Carlo (MCMC) and variational inference (VI), leveraging the advantages of both inference approaches. Specifically, we improve the variational distribution by running a few MCMC steps. To make inference tractable, we introduce the variational contrastive divergence (VCD)…

New statistical theory explains contrastive learning effectiveness.

VCL adds uncertainty to contrastive learning models.

Estimates CDF over complex regions using normalizing flows.

Self-supervised and supervised methods learn similar intermediate visual representations but diverge in final layers.

We study the geometry of probability distributions with respect to a generalized family of Csiszár $f$-divergences. A member of this family is the relative $α$-entropy which is also a Rényi analog of relative entropy in information theory and known as logarithmic or projective power divergence in statistics. We apply E…

Continuous-time PCD for MLE with explicit error bounds.

New theory explains GAN's high quality but low diversity.

Automatic detection of anomalies in space- and time-varying measurements is an important tool in several fields, e.g., fraud detection, climate analysis, or healthcare monitoring. We present an algorithm for detecting anomalous regions in multivariate spatio-temporal time-series, which allows for spotting the interesti…

The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demo…

Disagreement between two classifiers regarding the class membership of an observation in pattern recognition can be indicative of an anomaly and its nuance. As in general classifiers base their decision on class aposteriori probabilities, the most natural approach to detecting classifier incongruence is to use divergen…

Contrastive divergence (CD) is a promising method of inference in high dimensional distributions with intractable normalizing constants, however, the theoretical foundations justifying its use are somewhat shaky. This document proposes a framework for understanding CD inference, how/when it works, and provides multiple…

Estimating the log-likelihood gradient with respect to the parameters of a Restricted Boltzmann Machine (RBM) typically requires sampling using Markov Chain Monte Carlo (MCMC) techniques. To save computation time, the Markov chains are only run for a small number of steps, which leads to a biased estimate. This bias ca…

This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model. The divergence triangle is a compact and symmetric (anti-symmetric) objective function that seamlessly integrates variational learning, adversarial learning, wake-sleep algorithm, an…

LinMED is a new linear bandit algorithm with near-optimal regret bound.

Paper formalizes and analyzes a new bound for variational inference.

Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator …

Typical dimensionality reduction methods focus on directly reducing the number of random variables while retaining maximal variations in the data. In this paper, we consider the dimensionality reduction in parameter spaces of binary multivariate distributions. We propose a general Confident-Information-First (CIF) prin…

Fisher auto-encoders use Fisher divergence for more robust generative modeling.

New bounds found for optimizing non-convex functions with noisy data.

By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of th…

Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.

CD learning is shown to be an adversarial game for fitting models.

We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and sco…

Extends normalizing flows to arbitrary smooth manifolds.

Estimates causal effects using neural networks for balancing covariates.

Paper introduces a new method to improve GANs by leveraging the discriminator's energy.

Paper connects rejection learning to Bhattacharyya divergence.

RLMH improves adaptive MCMC by optimizing contrastive divergence reward.