This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
Paper introduces a simple method to assign uncertainty in contrastive learning models.
problem Contrastive learning models lack uncertainty measures.
method Trains a deep network to assign uncertainty based on representation variance.
result Deep uncertainty model improves anomaly detection and out-of-distribution classification.
Contrastive learning simplifies statistical inference for complex models.
problem Computational intractability of likelihood functions for certain models.
method Contrastive learning as an alternative for parameter estimation and inference.
result Contrastive learning enables practical methods for diverse statistical problems.
Contrastive learning helps linear models understand document topics.
problem Document classification with limited labeled data.
method Contrastive learning applied to document topic modeling.
result Linear models can recover topic posterior information from contrastive learning representations.
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
CUBE explains models by balanced experiments and contrasts.
problem Post-hoc explanation of trained predictive models.
method Design-based framework using balanced low-high probes.
result Reveals dominant learned effect structure and clarifies query efficiency.
Adaptive contrastive search improves text generation quality and diversity.
problem Decoding high-quality text from language model outputs.
method Adaptive contrastive search incorporating uncertainty-guided degeneration penalty.
result Enhanced text generation quality and diversity across different models and datasets.
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
Paper develops tighter risk certificates for contrastive learning models.
problem Statistical theory for contrastive learning is lacking, especially for practical models like SimCLR.
method Develops non-vacuous PAC-Bayesian risk certificates considering practical SimCLR factors.
result Risk certificates for contrastive loss and downstream prediction are much tighter than previous results.
Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.
problem Learning high-dimensional representations from multi-modal data.
method Multi-modal contrastive learning with temperature optimization.
result Contrastive learning adapts to intrinsic dimensions of data, not specified dimensions.
VCL adds uncertainty to contrastive learning models.
problem Lack of uncertainty quantification in contrastive learning methods.
method VCL uses a decoder-free framework that maximizes ELBO with InfoNCE loss and KL divergence.
result VCL provides meaningful uncertainty estimates and matches deterministic baselines in accuracy.
Improved unlearning in LLMs using contrastive decoding.
problem Removing specific information from LLMs while preserving overall performance.
method Inference-time unlearning algorithm using contrastive decoding with two auxiliary models.
result Notable gains in forget quality and retained performance compared to prior approaches.
New statistical theory explains contrastive learning effectiveness.
problem Understanding why contrastive learning works well for representation extraction.
method Developed a new theoretical framework based on approximate sufficient statistics.
result Near-sufficient encoders derived from contrastive learning can be adapted for downstream tasks.
Bayesian graph contrastive learning improves uncertainty quantification for graph analytics.
problem Uncertainty quantification for node representations in graph contrastive learning.
method Proposes a Bayesian framework to learn stochastic encoders representing nodes as distributions, providing uncertainty estimates.
result Significant improvement in performance on benchmark datasets compared to existing methods.
BrainSurfCNN predicts task contrasts from resting-state fingerprints, improving accuracy over baseline.
problem Predicting task-evoked activity from resting-state functional connectivity.
method Surface-based convolutional neural network (BrainSurfCNN) with reconstructive-contrastive loss.
result Significantly improved accuracy in predicting task contrasts over baseline.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
Contrastive learning works well with redundant data views.
problem Improving unsupervised learning with redundant data views.
method Theoretical analysis of contrastive learning in a multi-view setting.
result Linear functions of learned representations are nearly optimal on prediction tasks when views are redundant.
NCE and CD are shown to be equivalent ML methods.
problem Estimating unnormalised models without normalisation constant.
method NCE uses proxy criterion, CD uses importance sampling.
result NCE and CD are equivalent ML methods.
Unified view of contrastive and supervised learning improves model performance.
problem Improving model performance in classification, robustness, and detection.
method Hybrid discriminative-generative training of energy-based models.
result Improved performance on classification, robustness, out-of-distribution detection, and calibration.
CVRL tackles complex visual observations in reinforcement learning.
problem Complex visual observations in natural environments.
method Contrastive Variational Reinforcement Learning (CVRL) learns a contrastive variational model by maximizing mutual information between latent states and observations.
result CVRL achieves comparable performance with state-of-the-art model-based DRL methods and significantly outperforms them on tasks with complex observations.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
Poly-view contrastive learning improves image representation learning.
problem Learning representations from multiple related views in image data.
method Developed new representation learning objectives for poly-view tasks using information maximization and sufficient statistics.
result Poly-view contrastive models trained for fewer epochs and with smaller batch sizes outperform models trained for more epochs and with larger batch sizes.
A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.
problem Resource demands in contrastive learning schemes, especially with small-batch training.
method FlatNCE is a novel contrastive objective that no longer explicitly aims for a discriminative classification goal.
result FlatNCE enables immediate performance boost independent of subject-matter engineering efforts.
Theoretical study on how model architecture affects contrastive learning performance.
problem Understanding the role of model architecture in self-supervised learning.
method Theoretical analysis of contrastive learning, focusing on model capacity and clustering structures.
result Contrastive representations have lower dimensionality than the number of clusters in the data distribution.
GAEs are shown to be implicitly contrastive learners, revealing new design axes.
problem Understanding and improving graph autoencoders.
method Reinterpret GAEs as contrastive learners, identifying asymmetric views.
result Interpreting GAEs as contrastive learners offers clearer model understanding and design guidance.
CURL uses contrastive learning to improve reinforcement learning performance.
problem Improving reinforcement learning performance on complex tasks.
method Contrastive learning to extract high-level features from raw pixels, followed by off-policy control.
result CURL outperforms prior methods on DeepMind Control Suite and Atari Games.
Recently, a method [7] was proposed to generate contrastive explanations for differentiable models such as deep neural networks, where one has complete access to the model. In this work, we propose a method, Model Agnostic Contrastive Explanations Method (MACEM), to generate contrastive explanations for \emph{any} clas…
A new mutual information optimization method using self-supervised binary contrastive learning.
problem Improving self-supervised contrastive learning for better model performance.
method Proposes a novel loss function for contrastive learning that optimizes mutual information in positive and negative pairs.
result The proposed method outperforms state-of-the-art self-supervised contrastive frameworks on various benchmark datasets.
The paper proves probabilistic alignment between unseen modalities using contrastive learning.
problem Aligning unseen modalities in unsupervised learning.
method Bayesian approach and direct comparison of contrastive representations.
result Direct comparison of contrastive representations recovers the same likelihood ratio as probabilistic graphical models.
Replicated and validated Rank-N-Contrast for robust regression.
problem Deep regression models struggle with continuous sample orders.
method Contrastive learning of continuous representations by ranking samples.
result Improved performance and robustness of RNC framework.
CSI detects novelty by contrasting shifted instances, outperforming existing methods.
problem Detecting samples from outside the training distribution.
method Contrastive learning with distributionally shifted augmentations.
result CSI outperforms existing methods in various novelty detection scenarios.
Zero-shot contrastive loss improves text-guided image style transfer without extra training.
problem Stochastic nature of diffusion models leads to trade-offs between style transformation and content preservation.
method Proposes a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks.
result Method outperforms existing methods while preserving content and requiring no additional training.
New method uses contrastively trained GNNs for more reliable graph model evaluation.
problem Need effective methods to evaluate Graph Generative Models.
method Use representations from contrastively trained Graph Neural Networks (GNNs) for evaluation.
result Contrastively trained GNNs provide more reliable evaluation metrics than traditional or GNN-based approaches.
New attacks prevent both supervised and contrastive learning from private data.
problem Preventing unauthorized use of private data and commercial datasets.
method Contrastive-like data augmentations in supervised error minimization or maximization frameworks.
result Achieve state-of-the-art worst-case unlearnability across SL and CL algorithms.
Theoretical proof shows COMs are a type of contrastive divergence model with improved sampling.
problem Improving sampling quality in offline model-based optimization.
method Showed COMs are contrastive divergence models, proposed Langevin MCMC sampler, and decoupled model.
result Improved sampling quality achieved by decoupling model and using Langevin MCMC.
A new model cleans vocal note event annotations in music.
problem Erroneous labels in music datasets.
method Contrastive learning to automatically create local deformations of likely correct labels.
result Transcription model accuracy improves with the proposed strategy.
Improved unsupervised probing for ranking tasks using Contrast-Consistent Ranking.
problem Improving self-consistency in language model rankings.
method Adapting Contrast-Consistent Search (CCS) to Contrast-Consistent Ranking (CCR) for ranking tasks.
result CCR probing outperforms prompting techniques across different models and datasets.
Contrastive learning benefits from generated data but can be harmed by it too.
problem Contrastive learning's reliance on data augmentation and the impact of generated data.
method Investigates the role of generated data in contrastive learning and proposes Adaptive Inflation (AdaInf).
result Generated data can sometimes harm contrastive learning, and AdaInf improves performance.
Improves continual learning with theoretical guarantees and a new algorithm.
problem Learning incremental tasks with dynamic data distributions.
method Contrastive and distillation losses with theoretical performance guarantees.
result Theoretical performance bounds and improved continual learning performance.
ReCoRe learns invariant features for world navigation using contrastive learning and regularizers.
problem Limited sample efficiency and overfitting to training scenarios in RL for visual navigation.
method Contrastive unsupervised learning and intervention-invariant regularizer.
result Significantly improves sample efficiency and generalization in out-of-distribution point navigation tasks.
New theory explains contrastive learning via overlapping augmented views.
problem Lack of theoretical understanding of contrastive learning.
method Augmentation overlap perspective to improve downstream performance.
result Asymptotically closed bounds for downstream performance under weaker assumptions.
Filtering data with a pre-trained model improves multimodal contrastive learning performance.
problem Improving the quality of internet-scale multimodal datasets.
method Characterized the performance of filtered contrastive learning under a bimodal data generation model.
result Data filtering using a pre-trained model reduces contrastive learning error by a factor of η \sqrt{η} η in the large η η η regime. This work learns visual representations for deformable objects using contrastive estimation.
problem Challenges in learning plannable visual representations for deformable objects.
method Jointly optimizes visual representation and dynamics models using contrastive estimation.
result Substantial improvements in performance over standard model-based learning techniques.
Supervised contrastive learning improves image classification accuracy.
problem Improving image classification accuracy using supervised contrastive learning.
method Extending self-supervised batch contrastive approach to fully-supervised setting, leveraging label information.
result Top-1 accuracy of 81.4% on ImageNet dataset, outperforming cross-entropy.
Paper proposes a new regularization method to prevent model degradation under distribution shifts.
problem Model performance degrades under distribution shifts.
method Supervised contrastive learning with heterogeneous similarity.
result The proposed method outperforms existing regularization methods on benchmark datasets.
Contrastive learning estimates transition kernels for continuous-time stochastic processes.
problem Estimating transition kernels for continuous-time stochastic processes without labeled data.
method Contrastive learning applied to strong-mixing continuous-time stochastic processes.
result Contrastive learning can estimate transition kernels for small-to-mid-range intervals in the diffusion case.
CNPs improve function approximation by contrastive learning.
problem Learning from non-i.i.d function instantiations in high-dimensional, noisy spaces.
method CNPs with TCL and FCL contrastive branches for better function approximation.
result CNPs outperform other variants in function distribution reconstruction and parameter identification.
GCBS improves contrastive learning performance efficiently.
problem Efficiently mining hard negatives for contrastive learning.
method Global Contrastive Batch Sampling (GCBS) approximates batch assignment problem.
result GCBS improves state-of-the-art performance in sentence embedding and code-search tasks.