Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
Unified view on learning unnormalized distributions using NCE.
problem Learning unnormalized distributions across different communities.
method Noise-Contrastive Estimation (NCE) for a unified perspective.
result Established finite-sample convergence rates for exponential families.
Estimates unnormalized models with missing data using imputation and noise contrastive estimation.
problem Statistical models with intractable normalization constants and missing data.
method Combines imputation techniques with estimators for unnormalized models like noise contrastive estimation and score matching.
result Effective statistical inference with unnormalized models from missing data.
Meta-learning for estimating complex conditional distributions.
problem Estimating conditional densities in multimodal distributions.
method Noise contrastive estimation with kernel mean embeddings.
result Meta-learning can share representations across tasks for conditional density estimation.
Improved neural language models trained with dynamic noise-contrastive estimation.
problem Training large-scale language models efficiently and avoiding overfitting.
method Dynamic Noise-Contrastive Estimation (DNCE) to train neural trans-dimensional random field language models.
result DNCE reduces training cost and improves model performance on large datasets.
NCE and CD are shown to be equivalent ML methods.
problem Estimating unnormalised models without normalisation constant.
method NCE uses proxy criterion, CD uses importance sampling.
result NCE and CD are equivalent ML methods.
Calibration of simplified vine copulas using noise contrastive estimation
problem Modeling complex multivariate dependence structures
method Noise contrastive estimation for calibration
result Improved model accuracy when simplifying assumption is violated
Paper analyzes NCE method for unnormalized models, reducing asymptotic variance.
problem Estimating parameters of unnormalized models with high asymptotic variance.
method Proposes a method to reduce asymptotic variance by estimating auxiliary distribution parameters and analyzing objective function forms.
result NCE estimator is consistent and asymptotically normal, with reduced variance.
Contrastive learning performance doesn't degrade with more negative samples.
problem Theoretical and empirical evidence of negative samples hurting performance in contrastive learning.
method Simple theoretical setting and empirical support on CIFAR-10 and CIFAR-100 datasets.
result Contrastive learning performance does not degrade with the number of negative samples.
Researchers improve NCE by addressing its flat loss landscape issues.
problem NCE's poor performance due to an ill-behaved loss landscape.
method Introduced eNCE with an exponential loss and normalized gradient descent.
result Proven that landscape issues arise from inappropriate noise distribution.
Proposes NC-VAE to prevent posterior collapse in VAEs.
problem Posterior collapse in variational autoencoders (VAEs).
method Noise contrastive estimation applied to VAEs.
result Proves NC-VAE cannot reach posterior collapse and provides lower bounds.
Noise Contrastive Priors improve neural network uncertainty estimates.
problem Reliable uncertainty estimates for neural network predictions.
method Noise Contrastive Priors (NCPs) train models to output high uncertainty for data outside the training distribution.
result NCPs prevent overfitting outside the training distribution and yield useful uncertainty estimates.
New method estimates unnormalised models using semi-automated noise generation.
problem Estimating unnormalised models with intractable partition functions.
method Proposes a new method that leverages observed data for semi-automated noise generation.
result Improved performance compared to NCE when data lie in a lower-dimensional manifold.
NCVis speeds up data visualization for large datasets.
problem Performance issues in t-SNE for large datasets.
method Noise contrastive estimation for scalable visualization.
result NCVis outperforms state-of-the-art techniques in speed and quality.
New methods improve Monte Carlo estimation of partition functions.
problem Estimating the normalization constant of complex distributions.
method Annealing through paths of distributions to estimate partition functions.
result Optimal path for estimation is arithmetic, improving efficiency.
Improved neural TRF LMs for speech recognition with NCE and CNN integration.
problem Training inefficiency of neural TRF LMs on large training corpora.
method Reformulated TRFs, noise-contrastive estimation, CNN integration.
result Successful and efficient training on a 40x larger dataset with 1/3 training time and 4.7% WER reduction.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Improved VAE model enhances uncertainty estimation for out-of-distribution samples.
problem VAEs assign higher likelihood to out-of-distribution inputs.
method INCPVAE integrates noise contrastive prior into VAEs for reliable uncertainty estimation.
result INCPVAE outperforms standard VAEs in uncertainty estimation for OOD inputs.
VNCE improves parameter estimation and posterior inference for unnormalised models.
problem Learning parameters of unnormalised latent variable models is intractable.
method Variational noise-contrastive estimation (VNCE) using a variational lower bound to the NCE objective function.
result VNCE can be used for both parameter estimation and posterior inference of latent variables.
New insights into noise distribution for self-supervised learning.
problem Challenges the assumption that optimal noise should match data distribution.
method Turns to Noise-Contrastive Estimation (NCE) to define optimality of noise distribution.
result Optimal noise distribution is different from data distribution, challenging GANs assumption.
Improved VAEs by training a contrastive prior to match posterior.
problem Prior hole problem in VAEs, leading to poor image generation.
method Introduced a contrastive energy-based prior and trained it using noise contrastive estimation.
result Significant improvement in VAE generative performance on various datasets.
Adaptive multi-stage density ratio estimation improves learning of latent space EBM.
problem Learning energy-based models in latent space is computationally expensive and challenging.
method Adaptive multi-stage density ratio estimation using NCE to bridge the gap between prior and posterior densities.
result The method enables more expressive prior models and sharpens the latent space EBM.
Method estimates mixture models without normalization.
problem Estimating mixture models with intractable normalization.
method Extends noise contrastive estimation (NCE) for mixture models.
result Probabilistic clustering using deep representations.
We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and sco…
Replicated Softmax model, a well-known undirected topic model, is powerful in extracting semantic representations of documents. Traditional learning strategies such as Contrastive Divergence are very inefficient. This paper provides a novel estimator to speed up the learning based on Noise Contrastive Estimate, extende…
New findings show optimal noise in contrastive learning is not the same as data distribution.
problem The optimal noise distribution in contrastive learning is not the same as the data distribution.
method Empirical and theoretical analysis of contrastive learning methods.
result Deviation from the assumption of equal noise and data distribution leads to better statistical estimators.
Gaussians as noise in NCE lead to exponentially bad conditioning, hindering its efficiency.
problem Exponential conditioning of Hessian in NCE with Gaussian noise.
method Using Gaussian as the noise distribution in NCE.
result Gaussian noise in NCE leads to exponentially bad conditioning of the loss Hessian.
Self-supervised model detects phoneme boundaries without annotations.
problem Unsupervised phoneme segmentation without manual annotations.
method Convolutional neural network trained with Noise-Contrastive Estimation.
result Model outperforms baselines on TIMIT and Buckeye corpora.
New method tests causal association using noise contrastive backdoor adjustment.
problem Testing causal association in complex settings with many confounders.
method Backdoor-HSIC (bd-HSIC) using HSIC for independence testing.
result Calibrated and powerful for binary and continuous treatments with many confounders.
New method improves EBMs for regression tasks.
problem Training EBMs for regression is challenging.
method Proposed a simple yet effective extension of noise contrastive estimation.
result Our method achieves state-of-the-art performance on 1D regression and object detection.
MCD reformulates conditional density estimation into binary classification.
problem Conditional density estimation in statistical and machine learning.
method Marginal Contrastive Discrimination, reformulating into marginal and ratio density functions for binary classification.
result Significantly outperforms existing methods on most density models and regression datasets.
GANs with NCE improve dihedral angle prediction accuracy.
problem Inaccurate distribution of predicted dihedral angles.
method Introduced NCE-GAN to estimate density of GAN models and proposed residue-wise variants of AC-GAN and Semi-supervised GAN.
result Improved distribution of predicted angles, most similar to real angles with Semi-supervised GAN.
Developed criteria for selecting non-normalized models using NCE and score matching.
problem No information criteria for non-normalized models estimated by NCE or score matching.
method Developed information criteria based on discrepancy measures for non-normalized models estimated by NCE or score matching.
result The proposed criteria enable selection of the appropriate non-normalized model in a data-driven manner.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
CANE selects a subset of classes for large multi-class classification problems.
problem Large multi-class classification problems with high number of classes.
method CANE selects a small subset of candidate classes and samples the remaining classes.
result CANE achieves better prediction accuracy and significant speedup compared to state-of-the-art methods.
We present NN-grams, a novel, hybrid language model integrating n-grams and neural networks (NN) for speech recognition. The model takes as input both word histories as well as n-gram counts. Thus, it combines the memorization capacity and scalability of an n-gram model with the generalization ability of neural network…
EBM reduces dimensionality for estimating heterogeneous CATEs.
problem Estimating CATEs requires many confounding variables, increasing sample complexity.
method Proposes an EBM that learns a low-dimensional representation of variables.
result EBM representations keep CATE estimates consistent and perform better than other methods.
Adversarial dynamics embedding improves MLE of exponential family models.
problem Maximum likelihood estimation of exponential family models with neural network parametrization.
method Adversarial dynamics embedding to estimate the dual sampler and primal model simultaneously.
result Adversarial dynamics embedding leads to more effective learning and improved estimators compared to existing methods.
Paper trains two models to improve synthesis quality and unsupervised learning.
problem Improving synthesis quality and unsupervised learning.
method Joint training of an energy-based model and a flow-based model using adversarial value function.
result Significant improvement in synthesis quality of the flow model.
This paper broadens contrastive learning for disentangled representations without strict data distribution assumptions.
problem Learning disentangled representations from data with specific assumptions.
method Extends theoretical guarantees for disentanglement to a broader family of contrastive methods, relaxing data distribution assumptions.
result Identifiability of true latents for four contrastive losses proved without common independence assumptions.
Improved visual representation learning with conditional negative sampling.
problem Learning strong unsupervised visual representations using contrastive learning.
method Introduce a family of mutual information estimators that sample negatives conditionally.
result Improves accuracy by 2-5% points on four standard image datasets.
New dataset and analysis improve evaluation of visual representation models.
problem Insufficient evaluation methods for visual representation models.
method Analyzed five representations and developed a new dataset.
result Models with poor linear classification performance can still perform well on complex tasks.
Unified framework connects NCE, MIS, RLR, and bridge sampling for EBMs.
problem Challenges in parameter estimation for intractable likelihood EBMs.
method Unified framework connecting NCE, RLR, MIS, and bridge sampling.
result Unified perspective clarifies relationships among existing methods.
EBMs are flexible but hard to train; this paper explains methods.
problem Training Energy-Based Models is difficult due to the unknown normalizing constant.
method Explains MCMC, SM, and NCE for training EBMs, highlighting connections.
result Provides a friendly introduction to modern EBM training methods.
SuNCEt accelerates contrastive learning with minimal labeled data.
problem Efficiently learning visual representations with limited labeled data.
method Noise-contrastive estimation and neighbourhood component analysis-based semi-supervised loss.
result SuNCEt achieves semi-supervised learning accuracy with less than half the labeled data.
New method detects changes online with bounds on delay.
problem Detecting changes in data streams efficiently.
method Maximizes discrepancy between pre-change and post-change distributions.
result Non-asymptotic bounds on average running length and detection delay.
Adversarial training improves Word2Vec for basket completion tasks.
problem Improving Word2Vec for basket completion in recommendation systems.
method Adversarial Negative Sampling based on GANs to stabilize training.
result Significant performance improvements over standard Word2Vec models.
Contrary to standard statistical models, unnormalised statistical models only specify the likelihood function up to a constant. While such models are natural and popular, the lack of normalisation makes inference much more difficult. Here we show that inferring the parameters of a unnormalised model on a space Ω can …