Contrastive learning properties studied, including feature suppression and hierarchical learning.
problem Feature suppression and hierarchical learning in contrastive learning.
method Generalized contrastive loss, instance-based contrastive learning, explicit and controllable competing features.
result Contrastive learning can suppress and prevent the learning of competing features.
This work studies how contrastive learning extracts features from unlabeled data.
problem How neural networks trained by contrastive learning can extract features from unlabeled data.
method Formal analysis of contrastive learning's feature learning process, considering two types of features: sparse and dense.
result Contrastive learning using ReLU networks can learn sparse features if proper augmentations are adopted.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
Contrastive learning outperforms autoencoders and GANs in feature recovery and downstream tasks.
problem Theoretical understanding of contrastive learning's superiority in feature learning.
method Theoretical analysis of contrastive learning in linear representation settings.
result Contrastive learning outperforms autoencoders and GANs for feature recovery and in-domain downstream tasks.
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
Contrastive regularization improves semi-supervised learning by better propagating confident pseudo-labels.
problem Consistency regularization's limitation in high performance and efficiency.
method Proposes contrastive regularization to update model features, pushing confident labels into unlabeled samples.
result Improves semi-supervised learning tasks with fewer training iterations and robust performance.
CURL uses contrastive learning to improve reinforcement learning performance.
problem Improving reinforcement learning performance on complex tasks.
method Contrastive learning to extract high-level features from raw pixels, followed by off-policy control.
result CURL outperforms prior methods on DeepMind Control Suite and Atari Games.
Contrastive learning struggles with class collapse and feature suppression, revealing bias towards simpler solutions.
problem Contrastive learning struggles with class collapse and feature suppression, especially in supervised and unsupervised settings.
method Unified theoretical framework to determine which features are learnt by CL, revealing bias towards simpler solutions.
result Bias towards simpler solutions is a key factor in class collapse and feature suppression.
ExpCLR uses expert features to improve time-series representation learning.
problem Current representation learning approaches fail to ensure useful properties for time-series data.
method ExpCLR employs expert features to replace data transformations in contrastive learning, ensuring two useful properties for time-series representations.
result ExpCLR outperforms state-of-the-art methods on three real-world time-series datasets.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
New method proves non-contrastive self-supervised learning learns useful features.
problem Understanding how non-contrastive self-supervised learning (NS-SL) learns useful features.
method Proved in a linear network, NS-SL learns a desirable projection matrix and reduces sample complexity. Suggested weight decay acts as an implicit threshold.
result DirectCopy, a simpler and more efficient algorithm, outperforms DirectPred on various datasets.
Paper interprets contrastive learning dynamics using message passing.
problem Lack of rigorous understanding of contrastive learning dynamics.
method Casts contrastive objective into message passing scheme on augmentation graph.
result The learning dynamics of contrastive learning can be theoretically characterized.
Contrastive ICA identifies features in experimental groups relative to controls.
problem Jointly analyzing experimental and control datasets to identify salient features.
method Developed contrastive ICA (cICA) using tensor decomposition.
result cICA identifies patterns and visualizes data effectively, outperforming existing methods.
Feature normalization prevents collapse in non-contrastive learning dynamics.
problem Non-contrastive learning can collapse into a single point due to lack of repulsive force.
method Extended previous theory based on L2 loss to cosine loss, considering feature normalization.
result Cosine loss induces stable equilibrium, preventing collapse even with insufficient repulsive force.
Proposes Contrastive Clustering for improved clustering performance.
problem Improving clustering performance on various datasets.
method Instance- and cluster-level contrastive learning through data augmentations and feature space projections.
result Contrastive Clustering achieves significant improvements over 17 competitive methods.
NCL improves interpretability of deep features by enforcing non-negativity.
problem Lack of interpretability in deep representations.
method Non-negative Contrastive Learning (NCL) using non-negativity constraints.
result NCL outperforms standard contrastive learning in feature disentanglement and selection.
Dimensionality reduction (DR) is frequently used for analyzing and visualizing high-dimensional data as it provides a good first glance of the data. However, to interpret the DR result for gaining useful insights from the data, it would take additional analysis effort such as identifying clusters and understanding thei…
Paper presents a world model that learns invariant causal features using contrastive unsupervised learning.
problem Learning invariant causal features in unsupervised settings.
method Contrastive unsupervised learning with intervention invariant auxiliary task.
result Significantly outperforms state-of-the-art methods on out-of-distribution point navigation tasks.
Variational autoencoders are powerful algorithms for identifying dominant latent structure in a single dataset. In many applications, however, we are interested in modeling latent structure and variation that are enriched in a target dataset compared to some background---e.g. enriched in patients compared to the genera…
New method removes contrastive loss by adding a prediction head, revealing learning mechanisms.
problem Understanding why neural networks learn competitive representations despite trivial optima.
method Empirical and theoretical analysis of a trainable, identity-initialized prediction head.
result The trainable prediction head enables learning all features, preventing dimensional collapse.
A new asymmetric contrastive loss improves performance on imbalanced datasets.
problem Improving performance on imbalanced datasets using contrastive learning.
method Introducing an asymmetric contrastive loss (ACL) and asymmetric focal contrastive loss (AFCL).
result AFCL outperforms CL and FCL in terms of weighted and unweighted classification accuracies on imbalanced datasets.
Improves tabular data augmentation for contrastive learning.
problem Ineffective augmentation techniques for tabular data.
method Class-conditioned and feature-correlation based augmentation.
result Consistently outperforms conventional corruption methods.
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
A new method transfers adversarial robustness from teacher to student using feature distillation.
problem Adversarial robustness transfer across different models and tasks.
method Guided Adversarial Contrastive Distillation (GACD) with contrastive learning and sample reweighted estimation.
result GACD effectively transfers adversarial robustness from teacher to student, achieving comparable or better results.
SC-InfoNCE improves InfoNCE for feature clustering in contrastive learning.
problem Lack of theoretical understanding of InfoNCE's feature clustering mechanism.
method Introduced a transition probability matrix to model data augmentation dynamics and optimize feature similarity.
result SC-InfoNCE achieves strong performance across diverse domains, aligning feature similarity with downstream data.
Method enhances anomaly detection using contrastive learning and out-of-distribution data.
problem Improving anomaly detection in datasets with limited out-of-distribution data.
method Proposes a contrastive learning method that incorporates out-of-distribution data to enhance anomaly detection performance.
result The method significantly improves anomaly detection performance, even with limited out-of-distribution data.
Recent unsupervised approaches to domain adaptation primarily focus on minimizing the gap between the source and the target domains through refining the feature generator, in order to learn a better alignment between the two domains. This minimization can be achieved via a domain classifier to detect target-domain feat…
GraphCL learns node representations by maximizing similarity between perturbed node features.
problem Learning node representations in graph data without labeled data.
method Contrastive learning of node embeddings using graph neural networks and a loss function.
result Significantly outperforms state-of-the-art in unsupervised node classification benchmarks.
Paper proposes a method to extract style features from unlabeled data.
problem Extracting fine-grained features like styles from unlabeled data.
method Contrastive conditioned variational autoencoders with mutual information constraints.
result The method efficiently extracts style features from real-world natural image datasets.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of rea…
PCA++ improves robustness to background noise in contrastive learning.
problem Recovering shared signal subspaces from positive pairs in high-dimensional data with structured background noise.
method PCA++ uses hard uniformity-constrained contrastive learning to enforce identity covariance on projected features.
result PCA++ outperforms standard PCA and alignment-only PCA+ in simulations and real-world datasets.
Framework explains how dual deep networks learn features from unlabeled data.
problem Understanding self-supervised learning with dual deep networks.
method Theoretical framework and hierarchical latent tree model.
result Deep ReLU networks learn latent variables through contrastive SSL.
Extends conformal prediction to contrastive learning for better coverage of positive samples.
problem Lack of principled guarantees on coverage in contrastive learning.
method Introduces minimum-volume covering sets with learnable constraints.
result Improves inclusion-exclusion trade-offs in positive and negative samples.
RényiCL uses Rényi divergence for robust contrastive learning with stronger data augmentations.
problem Learning useful representations from multiple data views with hard augmentations.
method RényiCL employs Rényi divergence for contrastive learning, using a novel variational objective to manage hard negative sampling.
result RényiCL achieves better performance with stronger augmentations compared to other methods.
A new method learns outcome-aware spectral features for causal effect estimation.
problem Estimation of causal effects in the presence of hidden confounders.
method Augmented Spectral Feature Learning framework that minimizes a contrastive loss derived from an augmented operator incorporating outcome information.
result Our method remains effective even under spectral misalignment.
A new contrastive learning method extracts asset embeddings from financial time series.
problem Extracting meaningful latent features from noisy financial data.
method Contrastive learning framework using hypothesis testing for positive and negative samples.
result Effective asset embeddings significantly outperform existing methods on financial tasks.
SFAVEL distills features from language models for fact verification without annotations.
problem Fact verification with semantically meaningful and compact features.
method Self-supervised pretraining using contrastive loss with language models.
result Achieved state-of-the-art results on FB15k-237 and FEVER.
Infinite hierarchical contrastive clustering identifies personal environments linked to health outcomes.
problem Identifying meaningful relationships between environmental features and health outcomes on an individual level.
method Contrastive clustering framework with stick-breaking prior and participant-specific prediction loss.
result Model effectively identifies distinct personal environments and groups them into meaningful types linked to health outcomes.
Recent advances in interpretable Machine Learning (iML) and eXplainable AI (XAI) construct explanations based on the importance of features in classification tasks. However, in a high-dimensional feature space this approach may become unfeasible without restraining the set of important features. We propose to utilize t…
ReCoRe learns invariant features for world navigation using contrastive learning and regularizers.
problem Limited sample efficiency and overfitting to training scenarios in RL for visual navigation.
method Contrastive unsupervised learning and intervention-invariant regularizer.
result Significantly improves sample efficiency and generalization in out-of-distribution point navigation tasks.
PROTOCOL tackles imbalanced multi-view clustering by enhancing contrastive learning.
problem Class imbalance in real-world multi-view data.
method PROTOCOL uses partial optimal transport to perceive and mitigate imbalance, enhancing contrastive learning.
result PROTOCOL significantly improves clustering performance on imbalanced multi-view data.
CUBE explains models by balanced experiments and contrasts.
problem Post-hoc explanation of trained predictive models.
method Design-based framework using balanced low-high probes.
result Reveals dominant learned effect structure and clarifies query efficiency.
Recently, a method [7] was proposed to generate contrastive explanations for differentiable models such as deep neural networks, where one has complete access to the model. In this work, we propose a method, Model Agnostic Contrastive Explanations Method (MACEM), to generate contrastive explanations for \emph{any} clas…
Letter analyzes training dynamics of a nonlinear contrastive learning model in high dimensions.
problem Understanding training dynamics of nonlinear contrastive learning models in high-dimensional settings.
method High-dimensional analysis using McKean-Vlasov PDEs and low-dimensional ODEs.
result The model's performance evolves according to specific ODEs, revealing features like feature learnability and noise effects.
Improves few-shot learning for hierarchical data using hyperbolic space.
problem Few-shot class-incremental learning for hierarchical data.
method Contrastive learning in hyperbolic space, Poincaré ball model, hyperbolic contrastive loss, maximum entropy distribution.
result Effective improvement of coarse and fine class accuracies in few-shot conditions.
Unsupervised segmentation learns features without labels, improving accuracy.
problem Discover and localize semantically meaningful categories in images without annotations.
method Separates feature learning from cluster compactification; distills unsupervised features into discrete semantic labels using a contrastive loss function.
result Significant improvement over prior state of the art on semantic segmentation challenges.
A new topology design improves zero-shot classification performance in contrastive learning.
problem Improving zero-shot classification performance in contrastive visual-textual alignment.
method Proposed an alternative topology design using multiple class tokens and an oblique manifold with negative inner product.
result Improves zero-shot classification performance by an average of 6.1%.