A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New findings show optimal noise in contrastive learning is not the same as data distribution.
NCE and CD are shown to be equivalent ML methods.
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
Unified view on learning unnormalized distributions using NCE.
Contrastive learning performance doesn't degrade with more negative samples.
New insights into noise distribution for self-supervised learning.
Calibration of simplified vine copulas using noise contrastive estimation
Researchers improve NCE by addressing its flat loss landscape issues.
Many parametric statistical models are not properly normalised and only specified up to an intractable partition function, which renders parameter estimation difficult. Examples of unnormalised models are Gibbs distributions, Markov random fields, and neural network models in unsupervised deep learning. In previous wor…
A new whole-sentence language model - neural trans-dimensional random field language model (neural TRF LM), where sentences are modeled as a collection of random fields, and the potential function is defined by a neural network, has been introduced and successfully trained by noise-contrastive estimation (NCE). In this…
PiNGDA learns beneficial noise for graph augmentation stability.
Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e…
There are many models, often called unnormalized models, whose normalizing constants are not calculated in closed form. Maximum likelihood estimation is not directly applicable to unnormalized models. Score matching, contrastive divergence method, pseudo-likelihood, Monte Carlo maximum likelihood, and noise contrastive…
Quantum algorithm reduces CVA risk-neutral expectation estimation costs.
Improved VAEs by training a contrastive prior to match posterior.
Paper proposes a new loss function for conditional models using soft targets.
Replicated Softmax model, a well-known undirected topic model, is powerful in extracting semantic representations of documents. Traditional learning strategies such as Contrastive Divergence are very inefficient. This paper provides a novel estimator to speed up the learning based on Noise Contrastive Estimate, extende…
New methods improve Monte Carlo estimation of partition functions.
Gaussians as noise in NCE lead to exponentially bad conditioning, hindering its efficiency.
MCD reformulates conditional density estimation into binary classification.
Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of an independent normal prior in weight space imposes relatively weak constr…
Unnormalised latent variable models are a broad and flexible class of statistical models. However, learning their parameters from data is intractable, and few estimation techniques are currently available for such models. To increase the number of techniques in our arsenal, we propose variational noise-contrastive esti…
PCA++ improves robustness to background noise in contrastive learning.
DACL tackles domain-specific contrastive learning by using Mixup noise.
We take steps towards understanding the "posterior collapse (PC)" difficulty in variational autoencoders (VAEs),~i.e. a degenerate optimum in which the latent codes become independent of their corresponding inputs. We rely on calculus of variations and theoretically explore a few popular VAE models, showing that PC alw…
AVICA estimates noise levels for better group ICA source recovery.
We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of…
Improved VAE model enhances uncertainty estimation for out-of-distribution samples.
This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive…
This paper broadens contrastive learning for disentangled representations without strict data distribution assumptions.
We consider classification in the presence of class-dependent asymmetric label noise with unknown noise probabilities. In this setting, identifiability conditions are known, but additional assumptions were shown to be required for finite sample rates, and so far only the parametric rate has been obtained. Assuming thes…
This paper proposes a method for multi-class classification problems, where the number of classes K is large. The method, referred to as Candidates vs. Noises Estimation (CANE), selects a small subset of candidate classes and samples the remaining classes. We show that CANE is always consistent and computationally effi…
Adaptive multi-stage density ratio estimation improves learning of latent space EBM.
We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and sco…
Self-supervised model detects phoneme boundaries without annotations.
We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks. We exploit the primal-dual view of the MLE with a kinetics augmented model to obtain an estimate associated with an adversarial dua…
We develop a general method for estimating a finite mixture of non-normalized models. Here, a non-normalized model is defined to be a parametric distribution with an intractable normalization constant. Existing methods for estimating non-normalized models without computing the normalization constant are not applicable …
SuNCEt accelerates contrastive learning with minimal labeled data.
Trans-dimensional random field language models (TRF LMs) where sentences are modeled as a collection of random fields, have shown close performance with LSTM LMs in speech recognition and are computationally more efficient in inference. However, the training efficiency of neural TRF LMs is not satisfactory, which limit…
Improved visual representation learning with conditional negative sampling.
New method tests causal association using noise contrastive backdoor adjustment.
We consider the problem of estimating the mean and covariance of a distribution from iid samples in , in the presence of an fraction of malicious noise; this is in contrast to much recent work where the noise itself is assumed to be from a distribution of known type. The agnostic problem includes many…
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of rea…
Estimates intrinsic dimension of data sets robustly to noise.
Letter analyzes training dynamics of a nonlinear contrastive learning model in high dimensions.
We solve matrix denoising with both row and column correlations, setting limits and designing optimal methods.
Strong inductive biases prevent harmless interpolation in overparameterized models.
Modern methods for data visualization via dimensionality reduction, such as t-SNE, usually have performance issues that prohibit their application to large amounts of high-dimensional data. In this work, we propose NCVis -- a high-performance dimensionality reduction method built on a sound statistical basis of noise c…