SimCLR pre-training improves CNN performance with fewer labels.
problem Learning with fewer labeled data.
method SimCLR contrastive learning method combined with supervised fine-tuning.
result SimCLR pre-training with supervised fine-tuning achieves almost optimal test loss with fewer labeled data.
Study compares memorization of SimCLR to supervised and random labels training.
problem Understanding memorization in contrastive learning.
method Investigated SimCLR's memorization properties compared to supervised and random labels training.
result SimCLR's memorization is similar to random labels training in terms of training object complexity distribution.
SinSim improves self-supervised learning by integrating optimal transport into contrastive learning.
problem Lack of explicit regularization in contrastive learning methods leads to suboptimal generalization.
method Integrates Sinkhorn regularization from optimal transport theory into SimCLR.
result SinSim outperforms SimCLR and other self-supervised methods on various datasets.
G-SimCLR improves unsupervised learning by clustering images into pseudo labels.
problem Improving unsupervised learning for image recognition.
method Proposes a method to cluster images into pseudo labels to batch images of the same category.
result Comparable performance enhancements on CIFAR10 and ImageNet datasets.
Paper develops tighter risk certificates for contrastive learning models.
problem Statistical theory for contrastive learning is lacking, especially for practical models like SimCLR.
method Develops non-vacuous PAC-Bayesian risk certificates considering practical SimCLR factors.
result Risk certificates for contrastive loss and downstream prediction are much tighter than previous results.
SogCLR uses small batch sizes for global contrastive learning, achieving similar performance to SimCLR.
problem Existing contrastive learning methods require large batch sizes or large feature dictionaries.
method SogCLR, a memory-efficient Stochastic Optimization algorithm for global contrastive learning.
result SogCLR with small batch sizes (e.g., 256) achieves similar performance to SimCLR with large batch sizes (e.g., 8192).
Unified framework for SSL methods linking contrastive and non-contrastive approaches.
problem Lack of theoretical foundations and design guidelines for SSL methods.
method Spectral manifold learning framework to unify SSL methods.
result Theoretical bridge between contrastive and non-contrastive methods.
Develops a new self-supervised learning method combining contrastive and non-contrastive approaches.
problem Leveraging unlabeled data for representation learning, especially with high variance and low batch sizes.
method Converts a contrastive method (Spectral Contrastive Loss) into a non-contrastive form (MINC loss) to reduce variance and mutual information.
result MINC loss consistently improves upon the Spectral Contrastive loss baseline in learning image representations.
New statistical theory explains contrastive learning effectiveness.
problem Understanding why contrastive learning works well for representation extraction.
method Developed a new theoretical framework based on approximate sufficient statistics.
result Near-sufficient encoders derived from contrastive learning can be adapted for downstream tasks.
This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn use…
Framework explains how dual deep networks learn features from unlabeled data.
problem Understanding self-supervised learning with dual deep networks.
method Theoretical framework and hierarchical latent tree model.
result Deep ReLU networks learn latent variables through contrastive SSL.
CLOCS uses contrastive learning to improve cardiac signal representations.
problem Lack of labelled data in cardiac signal analysis.
method Contrastive learning across space, time, and patients.
result CLOCS outperforms state-of-the-art methods and achieves strong generalization.
Poly-view contrastive learning improves image representation learning.
problem Learning representations from multiple related views in image data.
method Developed new representation learning objectives for poly-view tasks using information maximization and sufficient statistics.
result Poly-view contrastive models trained for fewer epochs and with smaller batch sizes outperform models trained for more epochs and with larger batch sizes.
SASSL improves self-supervised learning by preserving image structure.
problem Distorted augmented samples in self-supervised learning.
method Neural Style Transfer to decouple semantic and stylistic attributes.
result Boosts ImageNet top-1 accuracy by up to 2 percentage points.
Unified framework interprets SSL models, revealing biases.
problem Interpreting black-box SSL models for transparency.
method KREPES framework using Representer Landmarks and Nyström approximation.
result Direct audit of SSL latent space without supervision.
VCL adds uncertainty to contrastive learning models.
problem Lack of uncertainty quantification in contrastive learning methods.
method VCL uses a decoder-free framework that maximizes ELBO with InfoNCE loss and KL divergence.
result VCL provides meaningful uncertainty estimates and matches deterministic baselines in accuracy.
DACL tackles domain-specific contrastive learning by using Mixup noise.
problem Domain-specific contrastive learning methods rely on data augmentation techniques that require domain knowledge.
method DACL uses Mixup noise to create similar and dissimilar examples without domain-specific data augmentation.
result DACL outperforms other domain-agnostic noising methods and combines well with domain-specific methods.
A new method for representation and metric learning on manifolds boosts performance.
problem Improving representation and metric learning on complex, non-Euclidean data.
method Atlas-based manifold representation with a modified encoder and MMD loss.
result Substantial performance boost over baseline for low-dimensional encodings.
A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.
problem Resource demands in contrastive learning schemes, especially with small-batch training.
method FlatNCE is a novel contrastive objective that no longer explicitly aims for a discriminative classification goal.
result FlatNCE enables immediate performance boost independent of subject-matter engineering efforts.
New bound proves rationality helps generalization in self-supervised learning.
problem Proving generalization gap in self-supervised learning.
method Proving upper bound on generalization gap for classifiers using rationality and self-supervised representations.
result Generalization gap tends to zero if classifier complexity is low relative to number of samples.
Generative model explains self-supervised learning across various tasks.
problem Lack of theoretical understanding of self-supervised learning methods.
method Generative latent variable model for self-supervised learning.
result Improves representation learning performance and narrows the gap between generative and discriminative methods.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
Contrastive learning benefits from generated data but can be harmed by it too.
problem Contrastive learning's reliance on data augmentation and the impact of generated data.
method Investigates the role of generated data in contrastive learning and proposes Adaptive Inflation (AdaInf).
result Generated data can sometimes harm contrastive learning, and AdaInf improves performance.
SAVeD detects dataset versions without metadata, improving accuracy and separation.
problem Difficulty in identifying similar versions of structured datasets.
method Contrastive learning with modified SimCLR pipeline, generating and contrasting augmented table views.
result SAVeD achieves higher accuracy and separation scores on unseen tables.
Optimizes contrastive learning with individualized temperatures for better performance on imbalanced datasets.
problem The common practice of using a global temperature parameter ignores the varying semantic similarity across different anchor data.
method Proposes a new robust contrastive loss inspired by distributionally robust optimization (DRO) and an efficient stochastic algorithm for automatic temperature individualization.
result Our method automatically learns a suitable temperature for each sample, improving performance on imbalanced datasets.
Adapts pretrained models to new classes without additional training.
problem Training models on limited labels and predicting new, unseen classes.
method Uses Fréchet mean instead of argmax for prediction, leveraging metric space distances.
result Improves model performance on unseen classes by up to 29.7% on ImageNet.
This study investigates self-supervised learning with Wasserstein distance on tree structures.
problem Improving self-supervised learning methods using Wasserstein distance.
method Utilized Tree-Wasserstein distance (TWD) and Jeffrey divergence regularization for training.
result A simple combination of softmax function and Tree-Wasserstein distance outperforms cosine similarity-based methods.
Novel bio-inspired masking for robust speech emotion recognition.
problem Noise degradation in speech emotion recognition.
method Cochlear cepstrogram-based contrastive learning with temporal and frequency masking.
result Improved speech emotion recognition performance on K-EmoCon benchmark.
New method improves unsupervised feature learning for natural data.
problem Natural data's correlated and long-tail distribution challenges instance-level contrastive learning.
method Cross-level instance-group discrimination (CLD) to integrate between-instance similarity.
result CLD achieves new state-of-the-art performance on various datasets.
Framework adds invariance to pretrained networks without fine-tuning.
problem Adding invariance to pretrained networks without altering original behavior.
method Post-training augmentation invariance framework with Markov-Wasserstein minimization and Wasserstein correlation maximization losses.
result Adapter networks improve classification accuracy on rotated and noisy images.
Unified principle LZN unifies generative modeling, representation learning, and classification.
problem Disjoint solutions for generative modeling, representation learning, and classification.
method LZN creates a shared latent space for all tasks, using encoders and decoders for each data type.
result LZN improves FID on CIFAR10 and outperforms existing methods in representation learning and joint generation/classification.