Supervised contrastive learning improves image classification accuracy.
problem Improving image classification accuracy using supervised contrastive learning.
method Extending self-supervised batch contrastive approach to fully-supervised setting, leveraging label information.
result Top-1 accuracy of 81.4% on ImageNet dataset, outperforming cross-entropy.
Proposes a new contrastive loss for semi-supervised medical image segmentation.
problem Lack of labeled data for medical image segmentation.
method Uses pseudo-labels and a local contrastive loss to learn good local representations.
result Achieved high segmentation performance on public cardiac and prostate datasets.
This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.
problem The question of whether there are fundamental differences in representation geometry between supervised contrastive learning and cross-entropy.
method The authors prove that both losses attain their minimum when representations of each class collapse to the vertices of a regular simplex, and they empirically validate this finding.
result Supervised contrastive learning requires more iterations to reach a close-to-optimal state compared to cross-entropy, indicating different optimization behavior.
Self-supervised method improves biosignal models with limited labeled data and subjects.
problem Limited labeled data and subjects in biosignals datasets.
method Contrastive learning with subject-aware loss and data augmentation.
result Self-supervised embeddings yield competitive results compared to supervised methods.
SoftCLT improves time series representation learning by soft contrastive loss.
problem Ignoring inherent correlations in time series leads to poor representation quality.
method SoftCLT introduces instance-wise and temporal contrastive loss with soft assignments.
result SoftCLT consistently improves various downstream tasks in time series learning.
This paper analyzes the landscape of supervised contrastive loss in over-parameterized networks.
problem Understanding the structure of solutions in over-parameterized networks under supervised contrastive loss.
method Analytical approach using unconstrained features model (UFM) to study the solutions of SC loss minimization.
result All local minima of SC loss are global minima in over-parameterized networks, and the minimizer is unique (up to rotation).
A new mutual information optimization method using self-supervised binary contrastive learning.
problem Improving self-supervised contrastive learning for better model performance.
method Proposes a novel loss function for contrastive learning that optimizes mutual information in positive and negative pairs.
result The proposed method outperforms state-of-the-art self-supervised contrastive frameworks on various benchmark datasets.
Paper proposes a novel GCN-based SSL algorithm to enhance node representations using contrastive and generative losses.
problem Shortage of supervision in graph-based semi-supervised learning.
method Combines contrastive and generative graph convolutional networks to enrich supervision signals.
result Improves node representations and classification results on various real-world datasets.
Develops a new self-supervised learning method combining contrastive and non-contrastive approaches.
problem Leveraging unlabeled data for representation learning, especially with high variance and low batch sizes.
method Converts a contrastive method (Spectral Contrastive Loss) into a non-contrastive form (MINC loss) to reduce variance and mutual information.
result MINC loss consistently improves upon the Spectral Contrastive loss baseline in learning image representations.
SuNCEt accelerates contrastive learning with minimal labeled data.
problem Efficiently learning visual representations with limited labeled data.
method Noise-contrastive estimation and neighbourhood component analysis-based semi-supervised loss.
result SuNCEt achieves semi-supervised learning accuracy with less than half the labeled data.
Unified framework for semi-supervised learning reduces annotation needs.
problem Sparse annotations and large amounts of unlabeled data in computational pathology.
method S5CL integrates fully-supervised, self-supervised, and semi-supervised learning through hierarchical contrastive losses.
result S5CL improves accuracy and F1-score in histopathological datasets with sparse labels.
Framework explains how dual deep networks learn features from unlabeled data.
problem Understanding self-supervised learning with dual deep networks.
method Theoretical framework and hierarchical latent tree model.
result Deep ReLU networks learn latent variables through contrastive SSL.
CLAMP uses neural manifold packing to improve self-supervised learning.
problem Improving self-supervised learning for vision tasks.
method CLAMP recasts representation learning as a manifold packing problem, introducing a loss function inspired by particle systems.
result CLAMP achieves competitive performance with state-of-the-art models and separates neural manifolds effectively.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Optimizes contrastive learning with individualized temperatures for better performance on imbalanced datasets.
problem The common practice of using a global temperature parameter ignores the varying semantic similarity across different anchor data.
method Proposes a new robust contrastive loss inspired by distributionally robust optimization (DRO) and an efficient stochastic algorithm for automatic temperature individualization.
result Our method automatically learns a suitable temperature for each sample, improving performance on imbalanced datasets.
Proves accuracy guarantees for self-supervised learning with correlated positive pairs.
problem Lack of theoretical guarantees for self-supervised learning with correlated positive pairs.
method Novel augmentation graph concept and spectral decomposition loss.
result Provably accurate features under linear probe evaluation.
New method improves self-supervised representation learning using probabilistic modeling and Monte Carlo integration.
problem Improving self-supervised representation learning for multimodal data.
method Discriminative probabilistic modeling with multiple importance sampling (MIS) for robust Monte Carlo integration.
result Proposes a novel non-parametric method for approximating conditional probability densities through convex optimization.
A new SSL method using whitening of latent-space features.
problem Efficiency and effectiveness of self-supervised representation learning.
method Proposes a new loss function based on whitening of latent-space features, avoiding the need for negatives and asymmetric networks.
result Improves efficiency and effectiveness of SSL by avoiding the need for negatives and asymmetric networks.
C2VAE learns disentangled and coupled representations without prior knowledge.
problem Learning disentangled and coupled representations in latent space.
method Introduces C2VAE, a self-supervised VAE that factorizes posterior and uses Gaussian copula for dependencies. result Demonstrates strong effect in enhancing disentangled representation learning.
Unified view of contrastive and supervised learning improves model performance.
problem Improving model performance in classification, robustness, and detection.
method Hybrid discriminative-generative training of energy-based models.
result Improved performance on classification, robustness, out-of-distribution detection, and calibration.
A simple framework improves visual representation learning.
problem Efficiently learning useful visual representations from unlabeled data.
method Contrastive learning with simplified framework and systematic study of components.
result Significant improvement in accuracy compared to previous methods.
A method for self-supervised learning of multivariate time series data across varying channels.
problem Labeling multivariate biomedical time series data is laborious and expensive.
method Proposes a multi-view self-supervised learning approach using a message passing neural network to extract a single representation across channels.
result Our method, combined with the TS2Vec loss, outperforms all other methods in most settings.
This paper finds ReLU restores symmetry in SCL under class imbalances.
problem Symmetry break in SCL under class imbalances.
method Analytical proof and experiments with ReLU activation and batch selection.
result ReLU restores symmetry in SCL-learned representations without loss in test accuracy.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
New measure quantifies contrastive self-supervised learning's generalization ability.
problem Limited theoretical understanding of contrastive self-supervised learning's generalization.
method Defined (σ,δ)-measure to mathematically quantify data augmentation and provide an upper bound for downstream classification error. result Generalization ability is related to alignment of positive samples, divergence of class centers, and concentration of augmented data.
Two CSSL-based methods improve graph classification with limited labeled data.
problem Limited labeled data for graph classification leads to overfitting.
method Contrastive self-supervised learning (CSSL) for graph encoders pretraining and regularization.
result CSSL methods reduce overfitting and improve graph classification accuracy.
New method improves transfer and robustness of supervised contrastive learning.
problem Class collapse in supervised contrastive learning leads to poor representation quality.
method Adding a weighted class-conditional InfoNCE loss and a class-conditional autoencoder.
result Improves transfer and robustness on 5 standard datasets and 3 worst-group robustness datasets.
GraphCL learns node representations by maximizing similarity between perturbed node features.
problem Learning node representations in graph data without labeled data.
method Contrastive learning of node embeddings using graph neural networks and a loss function.
result Significantly outperforms state-of-the-art in unsupervised node classification benchmarks.
Improved 3D MRI classification using contrastive learning with continuous proxy metadata.
problem Insufficient labelled data for 3D medical image classification.
method Proposed a new loss function (y-Aware InfoNCE) to leverage continuous proxy metadata in contrastive learning.
result 3D CNN model pre-trained on 10^4 multi-site healthy brain MRI scans outperforms fully-supervised methods.
New method improves model robustness to biased data.
problem Learning unbiased models from biased datasets.
method Developed epsilon-SupInfoNCE and FairKL losses.
result Improved performance on biased datasets.
Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.
problem Downstream task performances of GNN SSL on RDBs are poor.
method Proposed InfoNode to maximize mutual information between initial and final node representations.
result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.
Theoretical study on how model architecture affects contrastive learning performance.
problem Understanding the role of model architecture in self-supervised learning.
method Theoretical analysis of contrastive learning, focusing on model capacity and clustering structures.
result Contrastive representations have lower dimensionality than the number of clusters in the data distribution.
We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS3L is motivated by the success of negative sampling/contrastive estimation. We demonstrate that adding the NS3L loss to state-of-the-art SSL algorithms, such as the Virtual Adv…
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
FedMCC learns from distributed data to cluster and extract features.
problem Learning from distributed data and clustering.
method Federated Momentum Contrastive Clustering (FedMCC) framework.
result FedMCC outperforms existing methods in linear evaluation and semi-supervised learning.
SinSim improves self-supervised learning by integrating optimal transport into contrastive learning.
problem Lack of explicit regularization in contrastive learning methods leads to suboptimal generalization.
method Integrates Sinkhorn regularization from optimal transport theory into SimCLR.
result SinSim outperforms SimCLR and other self-supervised methods on various datasets.
DNN-based cross-modal retrieval has become a research hotspot, by which users can search results across various modalities like image and text. However, existing methods mainly focus on the pairwise correlation and reconstruction error of labeled data. They ignore the semantically similar and dissimilar constraints bet…
This work improves medical image segmentation with limited annotations using contrastive learning.
problem Lack of labeled data for medical image segmentation.
method Contrastive learning framework for semi-supervised segmentation with domain-specific and problem-specific cues.
result Significant improvements in segmentation performance compared to other methods.
Paper proposes a new regularization method to prevent model degradation under distribution shifts.
problem Model performance degrades under distribution shifts.
method Supervised contrastive learning with heterogeneous similarity.
result The proposed method outperforms existing regularization methods on benchmark datasets.
SogCLR uses small batch sizes for global contrastive learning, achieving similar performance to SimCLR.
problem Existing contrastive learning methods require large batch sizes or large feature dictionaries.
method SogCLR, a memory-efficient Stochastic Optimization algorithm for global contrastive learning.
result SogCLR with small batch sizes (e.g., 256) achieves similar performance to SimCLR with large batch sizes (e.g., 8192).
A new framework learns differentiable structured losses from data.
problem Learning effective losses for complex structured prediction tasks.
method Contrastive learning to learn differentiable structured losses from output data.
result Achieves similar or better performance than kernel-based methods.
Subg-Con learns graph representations from subgraphs, improving scalability and efficiency.
problem Scalability issues and weak supervision in graph representation learning.
method Subg-Con uses subgraphs sampled from the original graph to define a contrastive loss, learning node representations without complete graph data.
result Subg-Con outperforms existing methods in scalability, efficiency, and weak supervision requirements.
A new supervised tree-Wasserstein distance improves document classification.
problem Measuring document similarity efficiently and accurately.
method Rewriting Wasserstein distance on tree metric, using contrastive loss for optimization.
result The Supervised Tree-Wasserstein (STW) distance improves document classification accuracy.
Paper proposes a method to train robust neural networks without labeled data.
problem Training robust neural networks without class labels.
method Adversarial contrastive learning framework using unlabeled data.
result Robust Contrastive Learning (RoCL) achieves comparable robust accuracy to supervised methods and significantly improved robustness.
Kernel-based SSL creates useful representations without labels.
problem Creating useful representations without labels using self-supervised learning.
method Derive methods for kernel-based SSL, focusing on contrastive and non-contrastive loss functions.
result Kernel-induced representations correlate related points and de-correlate unrelated ones.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
New method removes contrastive loss by adding a prediction head, revealing learning mechanisms.
problem Understanding why neural networks learn competitive representations despite trivial optima.
method Empirical and theoretical analysis of a trainable, identity-initialized prediction head.
result The trainable prediction head enables learning all features, preventing dimensional collapse.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.