Contrastive learning helps linear models understand document topics.
problem Document classification with limited labeled data.
method Contrastive learning applied to document topic modeling.
result Linear models can recover topic posterior information from contrastive learning representations.
Efficient PAC learning for contrastive linear representations is achieved.
problem Efficient PAC learning for contrastive linear representations.
method Relaxing the problem to a semi-definite program and using Rademacher complexity.
result First efficient PAC learning algorithm for contrastive learning.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
Contrastive learning works well with redundant data views.
problem Improving unsupervised learning with redundant data views.
method Theoretical analysis of contrastive learning in a multi-view setting.
result Linear functions of learned representations are nearly optimal on prediction tasks when views are redundant.
Contrastive learning outperforms autoencoders and GANs in feature recovery and downstream tasks.
problem Theoretical understanding of contrastive learning's superiority in feature learning.
method Theoretical analysis of contrastive learning in linear representation settings.
result Contrastive learning outperforms autoencoders and GANs for feature recovery and in-domain downstream tasks.
Contrastive ICA identifies features in experimental groups relative to controls.
problem Jointly analyzing experimental and control datasets to identify salient features.
method Developed contrastive ICA (cICA) using tensor decomposition.
result cICA identifies patterns and visualizes data effectively, outperforming existing methods.
This work defines idealized SSL representations and improves existing methods.
problem Unclear characteristics of SSL representations leading to high downstream accuracies.
method Characterized ideal properties and derived necessary and sufficient conditions.
result Improved SSL methods and derived new objectives for contrastive and non-contrastive learning.
SSL framework identifies non-linear systems without labeled data.
problem System identification in non-linear environments without labeled data.
method Dynamics contrastive learning framework.
result SSL can identify non-linear dynamics in latent space.
Proves accuracy guarantees for self-supervised learning with correlated positive pairs.
problem Lack of theoretical guarantees for self-supervised learning with correlated positive pairs.
method Novel augmentation graph concept and spectral decomposition loss.
result Provably accurate features under linear probe evaluation.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.
problem Downstream task performances of GNN SSL on RDBs are poor.
method Proposed InfoNode to maximize mutual information between initial and final node representations.
result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.
Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.
problem Learning high-dimensional representations from multi-modal data.
method Multi-modal contrastive learning with temperature optimization.
result Contrastive learning adapts to intrinsic dimensions of data, not specified dimensions.
This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.
problem The question of whether there are fundamental differences in representation geometry between supervised contrastive learning and cross-entropy.
method The authors prove that both losses attain their minimum when representations of each class collapse to the vertices of a regular simplex, and they empirically validate this finding.
result Supervised contrastive learning requires more iterations to reach a close-to-optimal state compared to cross-entropy, indicating different optimization behavior.
New statistical theory explains contrastive learning effectiveness.
problem Understanding why contrastive learning works well for representation extraction.
method Developed a new theoretical framework based on approximate sufficient statistics.
result Near-sufficient encoders derived from contrastive learning can be adapted for downstream tasks.
Fast algorithm solves BVPs in linear time with probabilistic uncertainty.
problem Solving boundary value problems efficiently and accurately.
method Gauss--Markov prior tailored to BVPs, linear-time computation.
result Probabilistic solution with linear time complexity and comparable quality.
CLOCS uses contrastive learning to improve cardiac signal representations.
problem Lack of labelled data in cardiac signal analysis.
method Contrastive learning across space, time, and patients.
result CLOCS outperforms state-of-the-art methods and achieves strong generalization.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
Gradient descent on deep linear CNNs converges to a penalty-based solution.
problem Understanding gradient descent convergence in deep linear convolutional networks.
method Gradient descent on full-width linear convolutional networks of varying depth.
result Gradient descent converges to a penalty-based solution, not the hard margin SVM solution.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
New approach learns graph representations by contrasting first-order neighbors and graph diffusion views.
problem Learning node and graph level representations from graph data.
method Self-supervised approach using contrastive learning of multi-scale encodings.
result Achieves state-of-the-art performance on 8 out of 8 benchmarks.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
Enhances CLIP's similarity computation using PMI's linear structure.
problem CLIP's similarity computation misses the optimal linear structure of PMI.
method KME-CLIP, utilizing inner product in a reproducing kernel Hilbert space.
result KME-CLIP approximates PMI with arbitrary accuracy and outperforms CLIP.
Contrastive learning properties studied, including feature suppression and hierarchical learning.
problem Feature suppression and hierarchical learning in contrastive learning.
method Generalized contrastive loss, instance-based contrastive learning, explicit and controllable competing features.
result Contrastive learning can suppress and prevent the learning of competing features.
A new residual bootstrap method for high-dimensional regression with near low-rank designs.
problem Distributional approximation of linear contrasts in high-dimensional regression with near low-rank designs.
method Proposes a modified residual bootstrap method for ridge regression in high-dimensional settings with near low-rank designs.
result The modified residual bootstrap consistently approximates the laws of linear contrasts in the specified high-dimensional setting.
CW-ICA improves on ANICA for non-linear source separation.
problem Non-linear source separation challenges with many applications.
method CW-ICA extends ANICA by using a simpler, closed-form optimization target.
result CW-ICA achieves comparable results to ANICA without adversarial training.
A simple framework improves visual representation learning.
problem Efficiently learning useful visual representations from unlabeled data.
method Contrastive learning with simplified framework and systematic study of components.
result Significant improvement in accuracy compared to previous methods.
FedMCC learns from distributed data to cluster and extract features.
problem Learning from distributed data and clustering.
method Federated Momentum Contrastive Clustering (FedMCC) framework.
result FedMCC outperforms existing methods in linear evaluation and semi-supervised learning.
A new mutual information optimization method using self-supervised binary contrastive learning.
problem Improving self-supervised contrastive learning for better model performance.
method Proposes a novel loss function for contrastive learning that optimizes mutual information in positive and negative pairs.
result The proposed method outperforms state-of-the-art self-supervised contrastive frameworks on various benchmark datasets.
Deep learning improves breast cancer detection in DOT.
problem Complex physics and ill-posedness in DOT reconstruction.
method Deep learning approach that learns non-linear photon scattering physics.
result Deep neural network accurately recovers optical anomalies.
A new framework for interpretable models using sparse linear layers.
problem Performance degradation and lower interpretability in concept bottleneck models.
method Contrastive Language Image models and a single sparse linear layer with Bayesian inference.
result Our framework outperforms recent CBM approaches in accuracy and concept sparsity.
We present a generalization of independent component analysis (ICA), where instead of looking for a linear transform that makes the data components independent, we look for a transform that makes the data components well fit by a tree-structured graphical model. Treating the problem as a semiparametric statistical prob…
New methods improve feature extraction and representation quality in supervised and unsupervised DR.
problem Statistical dependence, data diversity, contrast, and interpretability in conventional DR methods.
method Combines linear and nonlinear formulations for three new independence criteria.
result Significant improvements in contrast, accuracy, and interpretability over baselines.
ACL improves robustness with unlabeled data, and we analyze its generalization using Rademacher complexity.
problem Improving robustness of deep networks against adversarial attacks using unlabeled data.
method We analyze the generalization performance of Adversarial Contrastive Learning (ACL) using Rademacher complexity.
result The average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task.
Extends linear representation hypothesis to categorical and hierarchical concepts in LLMs.
problem Representing concepts without natural contrasts in large language models.
method Formalizes linear representation hypothesis for categorical and hierarchical concepts, proving relationships between concept hierarchy and representation geometry.
result Validated theoretical results on large language models, estimating representations for 900+ concepts.
New inequality for regression risk with random design and noise.
problem Excess risk in least-squares regression with random design and heteroscedastic noise.
method Proved a new concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise, separating linearized and quadratic processes.
result Generalized the approach to quadratic contrasts and random design.
New method identifies causal relationships from interventions in complex systems.
problem Learning causal representations from unknown, latent interventions with general nonlinear mixing.
method Strong identifiability results with unknown single-node interventions, using geometric structure of transformed data.
result First instance of causal identifiability from non-paired interventions for deep neural network embeddings.
In this short note, we provide a sample complexity lower bound for learning linear predictors with respect to the squared loss. Our focus is on an agnostic setting, where no assumptions are made on the data distribution. This contrasts with standard results in the literature, which either make distributional assumption…
Study shows stability of Schwarzschild spacetime under specific perturbations.
problem Linear stability of Schwarzschild spacetime under axial perturbations.
method Complex line bundle interpretation and connection-level object analysis.
result Suitably regular initial data decay to a linearized Kerr metric.
Contrastive learning benefits from generated data but can be harmed by it too.
problem Contrastive learning's reliance on data augmentation and the impact of generated data.
method Investigates the role of generated data in contrastive learning and proposes Adaptive Inflation (AdaInf).
result Generated data can sometimes harm contrastive learning, and AdaInf improves performance.
New method proves non-contrastive self-supervised learning learns useful features.
problem Understanding how non-contrastive self-supervised learning (NS-SL) learns useful features.
method Proved in a linear network, NS-SL learns a desirable projection matrix and reduces sample complexity. Suggested weight decay acts as an implicit threshold.
result DirectCopy, a simpler and more efficient algorithm, outperforms DirectPred on various datasets.
cVAE enhances salient latent features using contrastive learning.
problem Identifying salient latent features in datasets with enriched variation.
method Contrastive Variational Autoencoder (cVAE) combining contrastive learning and deep generative models.
result cVAE effectively uncovers salient latent features across diverse datasets.
Filtering data with a pre-trained model improves multimodal contrastive learning performance.
problem Improving the quality of internet-scale multimodal datasets.
method Characterized the performance of filtered contrastive learning under a bimodal data generation model.
result Data filtering using a pre-trained model reduces contrastive learning error by a factor of η \sqrt{η} η in the large η η η regime. Paper introduces new loss functions for Siamese networks using FDA.
problem Training Siamese networks with improved loss functions.
method Proposes Fisher Discriminant Triplet (FDT) and Fisher Discriminant Contrastive (FDC) loss functions based on FDA.
result Shows effectiveness of FDT and FDC on MNIST and histopathology datasets.
CLAMP uses neural manifold packing to improve self-supervised learning.
problem Improving self-supervised learning for vision tasks.
method CLAMP recasts representation learning as a manifold packing problem, introducing a loss function inspired by particle systems.
result CLAMP achieves competitive performance with state-of-the-art models and separates neural manifolds effectively.
Efficient algorithm for unknown linear systems with convex costs.
problem Controlling an unknown linear system with stochastic convex costs.
method Optimism in the Face of Uncertainty paradigm.
result Achieves optimal T \sqrt{T} T regret-rate. This paper addresses measurement errors in high-dimensional compositional data using a log-contrast model calibration approach.
problem Measurement errors in high-dimensional regression models involving compositional covariates.
method Calibration approach for the linear log-contrast model under lenient sparsity conditions.
result Established asymptotic normality of the estimator for inference.
IKA approximates kernels with linear combinations of chosen functions, outperforming Nyström method.
problem Efficient kernel approximation for large datasets.
method IKA method approximates kernels as a linear combination of user-defined functions.
result IKA consistently outperformed Nyström method on the STL-10 dataset.
Kernel-based SSL creates useful representations without labels.
problem Creating useful representations without labels using self-supervised learning.
method Derive methods for kernel-based SSL, focusing on contrastive and non-contrastive loss functions.
result Kernel-induced representations correlate related points and de-correlate unrelated ones.