New unsupervised method selects hard negative samples for contrastive learning.
problem How to select good negative examples for contrastive learning without using true similarity information.
method Developed a new family of unsupervised sampling methods for hard negative selection.
result Improves downstream performance across multiple modalities.
SANS uses graph structure to find meaningful negatives for entity and relation embeddings.
problem Finding hard negatives for entity and relation embeddings in knowledge graphs.
method Structure Aware Negative Sampling (SANS) that selects negatives from a node's k-hop neighborhood.
result SANS finds semantically meaningful negatives and is competitive with state-of-the-art approaches.
New framework analyzes why more negative samples improve self-supervised learning performance.
problem Inconsistency between theoretical degradation and empirical improvement of downstream supervised tasks with more negative samples.
method Coupon collector's problem framework to analyze self-supervised representation learning with more negative samples.
result Bound can implicitly incorporate supervised loss in self-supervised loss by increasing negative samples.
New insights into negative sampling for graph representation learning.
problem Challenges in generating high-quality graph representations for large node sets.
method Theoretical analysis and derivation of negative sampling distribution correlation, proposing MCNS method.
result The negative sampling distribution should be positively but sub-linearly correlated to the positive sampling distribution.
A new method improves graph node embeddings by considering both nearby and distant node similarities.
problem Improving graph node embeddings by considering both nearby and distant node similarities.
method Distance-aware Negative Sampling (DNS) which maximizes cohesion at nearby node-pairs and separation at distant node-pairs.
result DNS outperforms baseline methods in downstream node classification tasks on various datasets and GRL algorithms.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skip…
Extends conformal prediction to contrastive learning for better coverage of positive samples.
problem Lack of principled guarantees on coverage in contrastive learning.
method Introduces minimum-volume covering sets with learnable constraints.
result Improves inclusion-exclusion trade-offs in positive and negative samples.
Contrastive learning performance doesn't degrade with more negative samples.
problem Theoretical and empirical evidence of negative samples hurting performance in contrastive learning.
method Simple theoretical setting and empirical support on CIFAR-10 and CIFAR-100 datasets.
result Contrastive learning performance does not degrade with the number of negative samples.
In recent years, the Word2Vec model trained with the Negative Sampling loss function has shown state-of-the-art results in a number of machine learning tasks, including language modeling tasks, such as word analogy and word similarity, and in recommendation tasks, through Prod2Vec, an extension that applies to modeling…
SCE improves network embedding using sparsest cut for negative samples only.
problem Efficient unsupervised network embedding using negative samples.
method Inspired by sparsest cut, SCE uses Laplacian smoothing for node representations.
result SCE outperforms GraphSAGE, G2G, and DGI in accuracy and scalability.
Improved visual representation learning with conditional negative sampling.
problem Learning strong unsupervised visual representations using contrastive learning.
method Introduce a family of mutual information estimators that sample negatives conditionally.
result Improves accuracy by 2-5% points on four standard image datasets.
New theory shows how learning algorithms can create a bias towards negative outcomes.
problem Negativity bias in adaptive learning algorithms.
method Generalization of the Hot Stove Effect to settings with negative estimates leading to smaller sample sizes.
result Negativity bias persists even when negative estimates do not lead to avoidance.
Proposes a novel method for generating hard negatives near time series data boundaries.
problem Challenges in generating effective negative samples for time series anomaly detection.
method Reconstruction-driven boundary negative generation framework using reinforcement learning.
result Improves anomaly representation learning and achieves competitive detection performance.
Improved estimation for imbalanced data using log odds correction and optimal sampling.
problem Parameter estimation with nonuniform negative sampling for imbalanced data.
method Derive asymptotic distribution of IPW estimator, derive optimal sampling probability, propose likelihood-based estimator.
result Improved estimator has the smallest asymptotic variance.
State-of-the-art approaches for Knowledge Base Completion (KBC) exploit deep neural networks trained with both false and true assertions: positive assertions are explicitly taken from the knowledge base, whereas negative ones are generated by random sampling of entities. In this paper, we argue that random sampling is …
Paper proposes an adversarial sampling method for efficient extreme classification.
problem Training classifiers over many classes is computationally expensive.
method Adversarial sampling to draw negative samples from an adversarial model.
result Significantly reduces training time by an order of magnitude.
Paper connects sampling and labeling biases in large-output spaces.
problem Efficient training in large-output spaces with label imbalance.
method Unified approach to address sampling and labeling biases.
result Different negative sampling schemes trade-off performance on dominant and rare labels.
Enhances contrastive learning for better representation learning on wild images.
problem Binary partition of views from same and different instances limits CL's performance.
method Doubly Contrastive Learning (CACR) with contrastive attraction and repulsion.
result CACR improves performance and robustness on wild image datasets.
Proposes a new method for posterior sampling using MMD with negative distance kernel.
problem Posterior sampling and conditional generative modeling.
method Approximates joint distribution using discrete Wasserstein gradient flows of MMD with negative distance kernel.
result Establishes an error bound for posterior distributions and proves the method is a Wasserstein gradient flow.
We show that the skip-gram formulation of word2vec trained with negative sampling is equivalent to a weighted logistic PCA. This connection allows us to better understand the objective, compare it to other word embedding methods, and extend it to higher dimensional models.
We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS3L is motivated by the success of negative sampling/contrastive estimation. We demonstrate that adding the NS3L loss to state-of-the-art SSL algorithms, such as the Virtual Adv…
Proposes a new method to improve regression models with reweighted samples.
problem Improves regression models' performance under low sample sizes and covariate perturbations.
method Reparametrizes sample weights using a doubly non-negative matrix and solves the reweighted estimate efficiently.
result Adversarial reweighting strategy delivers promising results on various datasets.
The paper introduces the concept of a cluster structure to define a joint distribution of the sample size and its exchangeable random partitions. The cluster structure allows the probability distribution of the random partitions of a subset of the sample to be dependent on the sample size, a feature not presented in a …
GCBS improves contrastive learning performance efficiently.
problem Efficiently mining hard negatives for contrastive learning.
method Global Contrastive Batch Sampling (GCBS) approximates batch assignment problem.
result GCBS improves state-of-the-art performance in sentence embedding and code-search tasks.
Adaptive sampler improves recommendation for implicit feedback data.
problem Challenges in predicting user preferences from implicit feedback data.
method Noisy-label robust learning for adaptive sampler design.
result Significant improvement in recommendation quality on real-world datasets.
A common approach to analyze a covariate-sample count matrix, an element of which represents how many times a covariate appears in a sample, is to factorize it under the Poisson likelihood. We show its limitation in capturing the tendency for a covariate present in a sample to both repeat itself and excite related ones…
New mutual information framework improves contrastive learning for vision tasks.
problem Maximizing mutual information for better unsupervised learning representations.
method Reformulated mutual information as a lower bound, introducing new negative sampling strategies.
result Improved representations outperform previous methods in various vision tasks.
Modern deep artificial neural networks have achieved great success in the domain of computer vision and beyond. However, their application to many real-world tasks is undermined by certain limitations, such as overconfident uncertainty estimates on out-of-distribution data or performance deterioration under data distri…
Unified framework for word embedding models using noise examples.
problem Improving word embedding models with negative sampling.
method Formulated a Word-Context Classification (WCC) framework that generalizes SkipGram word embedding models.
result The best noise distribution is the data distribution, improving both performance and training speed.
A novel GNN architecture improves link prediction by combining positive and negative samples.
problem Link prediction accuracy limited by node-wise methods and increased model complexity of edge-wise methods.
method Proposes a novel GNN architecture that uses both positive and negative samples to inform node embeddings.
result The method retains the inference speed of node-wise models while achieving competitive accuracy with edge-wise alternatives.
Novel neural likelihood ratio estimation for negative data in particle physics.
problem Estimating likelihood ratios with negative probability densities and weights.
method Introducing a novel loss function and a new model architecture based on signed mixture models.
result Demonstrated improved estimation on a real-world example from particle physics.
Efficient Bayesian variable selection for binomial and negative binomial data.
problem Computational challenges in Bayesian variable selection for complex models.
method Tempered Gibbs Sampling and MCMC scheme.
result Demonstrated effectiveness on cancer data with thousands of covariates.
We present a fast variational Bayesian algorithm for performing non-negative matrix factorisation and tri-factorisation. We show that our approach achieves faster convergence per iteration and timestep (wall-clock) than Gibbs sampling and non-probabilistic approaches, and do not require additional samples to estimate t…
Detection of protein-protein interactions (PPIs) plays a vital role in molecular biology. Particularly, infections are caused by the interactions of host and pathogen proteins. It is important to identify host-pathogen interactions (HPIs) to discover new drugs to counter infectious diseases. Conventional wet lab PPI pr…
Data analysis in high energy physics often deals with data samples consisting of a mixture of signal and background events. The sPlot technique is a common method to subtract the contribution of the background by assigning weights to events. Part of the weights are by design negative. Negative weights lead to the diver…
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservati…
Proposes a method to train classifiers with delayed feedback using a time window.
problem Training classifiers with delayed feedback that can be biased due to delayed user actions.
method Uses a time window to select samples for training, constructs unbiased empirical risk from all samples.
result Improves classifier performance by using all samples with a time window assumption.
Negative dependence improves machine learning performance.
problem Traditional independence limits machine learning capabilities.
method Negative dependence models, including DPPs and perturbed lattice models.
result Negative dependence often surpasses traditional methods in various machine learning tasks.
Debiased contrastive learning improves representation learning by correcting for same-label sampling.
problem Sampling negative examples from truly different labels improves performance in self-supervised representation learning.
method Developed a debiased contrastive objective that corrects for the sampling of same-label datapoints without true labels.
result The proposed debiased contrastive objective consistently outperforms state-of-the-art methods across vision, language, and reinforcement learning benchmarks.
Paper introduces negative margin loss for better few-shot classification accuracy.
problem Improving few-shot classification accuracy with metric learning.
method Introduces negative margin loss and analyzes its impact on feature discriminability.
result Negative margin loss outperforms regular softmax loss on few-shot classification benchmarks.
Non-Negative Matrix Factorization, NMF, attempts to find a number of archetypal response profiles, or parts, such that any sample profile in the dataset can be approximated by a close profile among these archetypes or a linear combination of these profiles. The non-negativity constraint is imposed while estimating arch…
Semi-supervised method enhances recommendation from sparse data.
problem Data sparsity in implicit feedbacks and lack of negative samples.
method Domain adaptation using textual features to align latent spaces.
result Our models significantly outperform existing models on real-world datasets.
A new boosting method reduces overfitting and negative transfer in transfer learning.
problem Overfitting and negative transfer in transfer learning.
method Importance sampling in boosting and random-forest based ensemble methodology.
result Performs better than competitive transfer learning methodologies 63% of the time.
News novelty predicts negative stock market returns.
problem Negative stock market returns due to increased news novelty.
method Quantified news novelty using entropy measure from recurrent neural network applied to a large news corpus.
result Entropy exposure carries a negative risk premium, indicating that assets positively correlated with entropy hedge aggregate news risk.
Self-taught learning is a technique that uses a large number of unlabeled data as source samples to improve the task performance on target samples. Compared with other transfer learning techniques, self-taught learning can be applied to a broader set of scenarios due to the loose restrictions on the source data. Howeve…
We consider the problem of estimating the parameters of a d-dimensional rectified Gaussian distribution from i.i.d. samples. A rectified Gaussian distribution is defined by passing a standard Gaussian distribution through a one-layer ReLU neural network. We give a simple algorithm to estimate the parameters (i.e., th…
New confidence intervals improve treatment effect estimation in randomized experiments.
problem Improving confidence intervals for treatment effects in randomized experiments.
method Systematic exploitation of negative dependence or variance adaptivity.
result Achieved nonasymptotic confidence intervals with the same effective sample size as asymptotic ones.