SC-InfoNCE improves InfoNCE for feature clustering in contrastive learning.
problem Lack of theoretical understanding of InfoNCE's feature clustering mechanism.
method Introduced a transition probability matrix to model data augmentation dynamics and optimize feature similarity.
result SC-InfoNCE achieves strong performance across diverse domains, aligning feature similarity with downstream data.
InfoNCE objective is equivalent to ELBO in RPM, linking to self-supervised learning.
problem Improving self-supervised learning methods by connecting them to variational inference.
method Recognizing RPM and showing InfoNCE as a simplified lower bound on MI, equal to ELBO in infinite sample limit.
result The actual InfoNCE objective is equal to the ELBO (up to a constant) in the infinite sample limit.
Paper refines InfoNCE for accurate mutual information estimation.
problem Indirect connection of InfoNCE to mutual information estimation.
method Introduces InfoNCE-anchor with an auxiliary anchor class for consistent density ratio estimation.
result InfoNCE-anchor yields a plug-in MI estimator with significantly reduced bias.
Improved similarity search in embeddings using InfoNCE loss.
problem Improving similarity search in embedding models trained by contrastive learning.
method Introduced a new continuity bound for InfoNCE loss via Gâteaux differentiation, preserving the averaging effect of negative samples.
result Demonstrated that the averaging effect of k negative samples in InfoNCE loss carries over to stabilisation of generalisation error as k grows. The paper analyzes the InfoNCE loss under different temperature schedules using Langevin dynamics.
problem Understanding the dynamics of InfoNCE loss under fixed versus annealed temperature schedules.
method Modeling embedding evolution under Langevin dynamics on a compact Riemannian manifold, with theoretical guarantees for convergence.
result Slow logarithmic inverse-temperature schedules ensure convergence to globally optimal representations, while faster schedules risk suboptimal minima.
WEINCE improves contrastive learning by correcting softmax biases.
problem Softmax in InfoNCE can lead to misaligned statistical assumptions in contrastive learning.
method WEINCE uses anchor-wise online batch statistics to blend softmax logits with an endpoint shortfall correction.
result WEINCE yields consistent improvements in frozen-feature evaluation across five vision benchmarks.
A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.
problem Resource demands in contrastive learning schemes, especially with small-batch training.
method FlatNCE is a novel contrastive objective that no longer explicitly aims for a discriminative classification goal.
result FlatNCE enables immediate performance boost independent of subject-matter engineering efforts.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
New method AnInfoNCE uncovers latent factors in contrastive learning with practical variability.
problem Theoretical assumptions of contrastive learning loss overlook practical variability in positive pairs.
method AnInfoNCE, a generalization of InfoNCE, models anisotropic variability to uncover latent factors.
result AnInfoNCE increases recovery of latent factors in CIFAR10 and ImageNet, albeit at the cost of accuracy.
Proposes a constraint for deep clustering to handle both simple and complex topologies.
problem Limited prior knowledge for deep clustering methods to perform well on complex topologies.
method Introduces a constraint using symmetric InfoNCE to enhance deep clustering performance.
result The constraint improves deep clustering methods' performance on both simple and complex topologies.
New method improves transfer and robustness of supervised contrastive learning.
problem Class collapse in supervised contrastive learning leads to poor representation quality.
method Adding a weighted class-conditional InfoNCE loss and a class-conditional autoencoder.
result Improves transfer and robustness on 5 standard datasets and 3 worst-group robustness datasets.
New method improves self-supervised representation learning using probabilistic modeling and Monte Carlo integration.
problem Improving self-supervised representation learning for multimodal data.
method Discriminative probabilistic modeling with multiple importance sampling (MIS) for robust Monte Carlo integration.
result Proposes a novel non-parametric method for approximating conditional probability densities through convex optimization.
cMIM improves representation learning without positive-pair augmentations.
problem Learning robust representations for diverse tasks.
method Contrastive Mutual Information Machine (cMIM) framework.
result cMIM outperforms MIM and InfoNCE on classification and regression tasks.
This paper broadens contrastive learning for disentangled representations without strict data distribution assumptions.
problem Learning disentangled representations from data with specific assumptions.
method Extends theoretical guarantees for disentanglement to a broader family of contrastive methods, relaxing data distribution assumptions.
result Identifiability of true latents for four contrastive losses proved without common independence assumptions.
We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal,…
A new contrastive MI estimator improves efficiency and tightness.
problem Efficient and tight mutual information estimation.
method Contrastive Fenchel-Legendre optimization.
result The FLO estimator is tight and converges under stochastic gradient descent.
VCL adds uncertainty to contrastive learning models.
problem Lack of uncertainty quantification in contrastive learning methods.
method VCL uses a decoder-free framework that maximizes ELBO with InfoNCE loss and KL divergence.
result VCL provides meaningful uncertainty estimates and matches deterministic baselines in accuracy.
New measure quantifies contrastive self-supervised learning's generalization ability.
problem Limited theoretical understanding of contrastive self-supervised learning's generalization.
method Defined (σ,δ)-measure to mathematically quantify data augmentation and provide an upper bound for downstream classification error. result Generalization ability is related to alignment of positive samples, divergence of class centers, and concentration of augmented data.
Contrastive learning recovers latent distributions for ambiguous inputs, including aleatoric uncertainty.
problem Real-world observations often have inherent ambiguities, making the true posterior probabilistic with heteroscedastic uncertainty.
method Extended InfoNCE objective and encoders to predict latent distributions, proving they recover the correct posteriors, including aleatoric uncertainty.
result Contrastive learning encoders can recover the correct posteriors of data-generating processes, including aleatoric uncertainty, up to a rotation of the latent space.
New method improves model robustness to biased data.
problem Learning unbiased models from biased datasets.
method Developed epsilon-SupInfoNCE and FairKL losses.
result Improved performance on biased datasets.
New mutual information framework improves contrastive learning for vision tasks.
problem Maximizing mutual information for better unsupervised learning representations.
method Reformulated mutual information as a lower bound, introducing new negative sampling strategies.
result Improved representations outperform previous methods in various vision tasks.
Framework explains how dual deep networks learn features from unlabeled data.
problem Understanding self-supervised learning with dual deep networks.
method Theoretical framework and hierarchical latent tree model.
result Deep ReLU networks learn latent variables through contrastive SSL.
Maximizes image representation dependence for self-supervised learning.
problem Learning meaningful image representations from unlabeled data.
method Maximizes Hilbert-Schmidt Independence Criterion (HSIC) between image transformations and identity.
result Matches state-of-the-art performance on ImageNet and other vision tasks.
Improved 3D MRI classification using contrastive learning with continuous proxy metadata.
problem Insufficient labelled data for 3D medical image classification.
method Proposed a new loss function (y-Aware InfoNCE) to leverage continuous proxy metadata in contrastive learning.
result 3D CNN model pre-trained on 10^4 multi-site healthy brain MRI scans outperforms fully-supervised methods.
This work improves independence tests for high-dimensional data.
problem Detecting subtle dependencies between high-dimensional random variables with complex distributions.
method Develops two approaches to learn powerful independence tests using variational mutual information and HSIC.
result Optimized HSIC tests generally outperform other approaches on detecting structured dependence.
New MI bounds improve estimation in deep generative models.
problem Estimating mutual information without density information is intractable.
method Importance sampling, Annealed Importance Sampling, Generalized IWAE, MINE-AIS.
result Improved bounds for estimating mutual information in deep models.