Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
We extend contrastive learning theory for multiway classification and prove convergence guarantees.
problem Efficient self-supervised training for multiway classification tasks.
method Contrastive representation learning with multiple negative samples and convergence guarantees for gradient descent.
result Convergence guarantees for contrastive learning with gradient descent of an overparametrized encoder.
SuNCEt accelerates contrastive learning with minimal labeled data.
problem Efficiently learning visual representations with limited labeled data.
method Noise-contrastive estimation and neighbourhood component analysis-based semi-supervised loss.
result SuNCEt achieves semi-supervised learning accuracy with less than half the labeled data.
End-to-end training solves deep unsupervised contrastive learning problems.
problem Theoretical analysis of unsupervised contrastive learning.
method End-to-end training of deep neural networks.
result Approximate stationary solutions found for non-convex contrastive loss.
A new contrastive learning objective, FlatNCE, fixes resource demands in small-batch training.
problem Resource demands in contrastive learning schemes, especially with small-batch training.
method FlatNCE is a novel contrastive objective that no longer explicitly aims for a discriminative classification goal.
result FlatNCE enables immediate performance boost independent of subject-matter engineering efforts.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
CSI detects novelty by contrasting shifted instances, outperforming existing methods.
problem Detecting samples from outside the training distribution.
method Contrastive learning with distributionally shifted augmentations.
result CSI outperforms existing methods in various novelty detection scenarios.
Combining self-training and contrastive learning improves performance under distribution shift.
problem Improving performance under distribution shift using unlabeled data.
method Combining self-training and contrastive learning techniques.
result Combined method achieves 3-8% higher accuracy than either approach independently.
Paper proposes contrastive training to improve OOD detection without needing explicit OOD examples.
problem Improving reliable detection of out-of-distribution inputs for machine learning systems.
method Contrastive training approach that doesn't require explicit OOD examples, using CLP score.
result Contrastive training significantly improves OOD detection performance on benchmarks, especially in near OOD classes.
GCBS improves contrastive learning performance efficiently.
problem Efficiently mining hard negatives for contrastive learning.
method Global Contrastive Batch Sampling (GCBS) approximates batch assignment problem.
result GCBS improves state-of-the-art performance in sentence embedding and code-search tasks.
Poly-view contrastive learning improves image representation learning.
problem Learning representations from multiple related views in image data.
method Developed new representation learning objectives for poly-view tasks using information maximization and sufficient statistics.
result Poly-view contrastive models trained for fewer epochs and with smaller batch sizes outperform models trained for more epochs and with larger batch sizes.
Study compares memorization of SimCLR to supervised and random labels training.
problem Understanding memorization in contrastive learning.
method Investigated SimCLR's memorization properties compared to supervised and random labels training.
result SimCLR's memorization is similar to random labels training in terms of training object complexity distribution.
Improves continual learning with theoretical guarantees and a new algorithm.
problem Learning incremental tasks with dynamic data distributions.
method Contrastive and distillation losses with theoretical performance guarantees.
result Theoretical performance bounds and improved continual learning performance.
CLOCS uses contrastive learning to improve cardiac signal representations.
problem Lack of labelled data in cardiac signal analysis.
method Contrastive learning across space, time, and patients.
result CLOCS outperforms state-of-the-art methods and achieves strong generalization.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
GCC pre-trains graph neural networks to transfer across diverse datasets.
problem Non-transferable graph models trained for specific datasets.
method Graph Contrastive Coding (GCC) framework using contrastive learning.
result GCC pre-trained models achieve competitive performance across multiple datasets.
Filtering data with a pre-trained model improves multimodal contrastive learning performance.
problem Improving the quality of internet-scale multimodal datasets.
method Characterized the performance of filtered contrastive learning under a bimodal data generation model.
result Data filtering using a pre-trained model reduces contrastive learning error by a factor of η \sqrt{η} η in the large η η η regime. Improved unlearning in LLMs using contrastive decoding.
problem Removing specific information from LLMs while preserving overall performance.
method Inference-time unlearning algorithm using contrastive decoding with two auxiliary models.
result Notable gains in forget quality and retained performance compared to prior approaches.
Unified view of contrastive and supervised learning improves model performance.
problem Improving model performance in classification, robustness, and detection.
method Hybrid discriminative-generative training of energy-based models.
result Improved performance on classification, robustness, out-of-distribution detection, and calibration.
A new model cleans vocal note event annotations in music.
problem Erroneous labels in music datasets.
method Contrastive learning to automatically create local deformations of likely correct labels.
result Transcription model accuracy improves with the proposed strategy.
Zero-shot contrastive loss improves text-guided image style transfer without extra training.
problem Stochastic nature of diffusion models leads to trade-offs between style transformation and content preservation.
method Proposes a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks.
result Method outperforms existing methods while preserving content and requiring no additional training.
A new topology design improves zero-shot classification performance in contrastive learning.
problem Improving zero-shot classification performance in contrastive visual-textual alignment.
method Proposed an alternative topology design using multiple class tokens and an oblique manifold with negative inner product.
result Improves zero-shot classification performance by an average of 6.1%.
This paper analyzes the landscape of supervised contrastive loss in over-parameterized networks.
problem Understanding the structure of solutions in over-parameterized networks under supervised contrastive loss.
method Analytical approach using unconstrained features model (UFM) to study the solutions of SC loss minimization.
result All local minima of SC loss are global minima in over-parameterized networks, and the minimizer is unique (up to rotation).
Contrastive embeddings improve neural architecture search performance.
problem Improving performance of neural architecture search algorithms.
method Contrastive learning to identify networks based on data Jacobians and produce embeddings.
result Traditional black-box optimization algorithms can reach state-of-the-art performance with contrastive embeddings.
Contrastive Code Representation Learning improves code summarization and type inference.
problem Code representations are sensitive to edits, hindering downstream semantic understanding tasks.
method ContraCode: a contrastive pre-training task that learns code functionality.
result Contrastive pre-training improves code summarization and type inference accuracy.
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
i-Mix improves contrastive learning across domains without domain-specific augmentations.
problem Improving contrastive representation learning for unlabeled data across diverse domains.
method i-Mix treats contrastive learning as a non-parametric classifier problem, mixing data in input and virtual label spaces.
result i-Mix consistently improves representation quality across image, speech, and tabular data domains.
This work improves medical image segmentation with limited annotations using contrastive learning.
problem Lack of labeled data for medical image segmentation.
method Contrastive learning framework for semi-supervised segmentation with domain-specific and problem-specific cues.
result Significant improvements in segmentation performance compared to other methods.
Contrastive learning estimates transition kernels for continuous-time stochastic processes.
problem Estimating transition kernels for continuous-time stochastic processes without labeled data.
method Contrastive learning applied to strong-mixing continuous-time stochastic processes.
result Contrastive learning can estimate transition kernels for small-to-mid-range intervals in the diffusion case.
ACL improves robustness with unlabeled data, and we analyze its generalization using Rademacher complexity.
problem Improving robustness of deep networks against adversarial attacks using unlabeled data.
method We analyze the generalization performance of Adversarial Contrastive Learning (ACL) using Rademacher complexity.
result The average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task.
Letter analyzes training dynamics of a nonlinear contrastive learning model in high dimensions.
problem Understanding training dynamics of nonlinear contrastive learning models in high-dimensional settings.
method High-dimensional analysis using McKean-Vlasov PDEs and low-dimensional ODEs.
result The model's performance evolves according to specific ODEs, revealing features like feature learnability and noise effects.
Contrastive regularization improves semi-supervised learning by better propagating confident pseudo-labels.
problem Consistency regularization's limitation in high performance and efficiency.
method Proposes contrastive regularization to update model features, pushing confident labels into unlabeled samples.
result Improves semi-supervised learning tasks with fewer training iterations and robust performance.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
The paper proves probabilistic alignment between unseen modalities using contrastive learning.
problem Aligning unseen modalities in unsupervised learning.
method Bayesian approach and direct comparison of contrastive representations.
result Direct comparison of contrastive representations recovers the same likelihood ratio as probabilistic graphical models.
Contrastive examples improve fairness in face recognition by balancing minority and majority groups.
problem Face recognition algorithms favor majority groups in training data.
method Create contrastive examples by swapping group memberships in the training dataset.
result Contrastive examples improve fairness metrics like equalized odds.
Improved VAEs by training a contrastive prior to match posterior.
problem Prior hole problem in VAEs, leading to poor image generation.
method Introduced a contrastive energy-based prior and trained it using noise contrastive estimation.
result Significant improvement in VAE generative performance on various datasets.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
New method uses contrastively trained GNNs for more reliable graph model evaluation.
problem Need effective methods to evaluate Graph Generative Models.
method Use representations from contrastively trained Graph Neural Networks (GNNs) for evaluation.
result Contrastively trained GNNs provide more reliable evaluation metrics than traditional or GNN-based approaches.
Unified model learns from proteins and ligands for drug design.
problem Disjoint data sources and modeling assumptions limit joint use of structure- and ligand-based drug design.
method Contrastive Geometric Learning for Unified Computational Drug Design (ConGLUDe)
result Unified model achieves competitive zero-shot virtual screening performance and state-of-the-art ligand-conditioned pocket selection.
This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn use…
CCM improves context for Meta-RL by contrastive learning.
problem Improving context for Meta-RL to enable task generalization.
method CCM framework using contrastive learning for context encoding and information-gain-based trajectory collection.
result CCM outperforms state-of-the-art algorithms in benchmarks and sparse-reward environments.
Contrastive learning helps linear models understand document topics.
problem Document classification with limited labeled data.
method Contrastive learning applied to document topic modeling.
result Linear models can recover topic posterior information from contrastive learning representations.
SAINT improves neural networks for tabular data with row attention and contrastive pre-training.
problem Tabular data challenges in machine learning applications.
method SAINT combines row and column attention with contrastive self-supervised pre-training.
result SAINT outperforms previous deep learning methods and even gradient boosting methods on benchmark tasks.
Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.
problem Learning high-dimensional representations from multi-modal data.
method Multi-modal contrastive learning with temperature optimization.
result Contrastive learning adapts to intrinsic dimensions of data, not specified dimensions.
This paper proposes a more efficient training method for energy-based models.
problem The computational burden and validity trade-off in Contrastive Divergence training.
method Introducing Diffusion Contrastive Divergence (DCD) to replace Langevin dynamics with diffusion processes.
result The proposed DCDs are more computationally efficient and handle gradient terms better than Contrastive Divergence.
CUBE explains models by balanced experiments and contrasts.
problem Post-hoc explanation of trained predictive models.
method Design-based framework using balanced low-high probes.
result Reveals dominant learned effect structure and clarifies query efficiency.
New findings suggest non-contrastive learning has many bad minima, not just collapsed ones.
problem The effectiveness of non-contrastive learning in unsupervised feature learning.
method Theoretical analysis and controlled experiments on simple data models.
result Non-contrastive losses have a preponderance of non-collapsed bad minima, and these minima are not avoided during training.
Study shows uncertainty of deep learning models can be measured from their embeddings.
problem Uncertainty in contrastive learning models for critical applications.
method Estimating the distribution of training data in embedding space and accounting for local consistency.
result Uncertainty of an embedding vector correlates strongly with downstream accuracy.