We explain SSL objectives as log-likelihoods in a data curation model.
problem Lack of understanding of SSL objectives as log-likelihoods.
method Formulate SSL objectives as a log-likelihood in a generative model of data curation.
result SSL methods can be understood as lower-bounds on a principled log-likelihood.
Paper analyzes how unlabeled data improves SSL and adversarial robustness.
problem Understanding how unlabeled data impacts SSL and adversarial robustness.
method Minimax analysis and adversarial training.
result Reconstruction-based SSL algorithm is rate-optimal under various models and enhances adversarial robustness.
A new machine learning methodology, called successive subspace learning (SSL), is introduced in this work. SSL contains four key ingredients: 1) successive near-to-far neighborhood expansion; 2) unsupervised dimension reduction via subspace approximation; 3) supervised dimension reduction via label-assisted regression …
Semi-supervised learning benefits the rich more than the poor, affecting fairness.
problem Disparate impact of semi-supervised learning on different sub-populations.
method Theoretical and empirical analysis of a broad family of SSL algorithms using pseudo-labels.
result Semi-supervised learning benefits the rich more than the poor, potentially violating fairness.
Researchers propose a new SSL risk decomposition method to evaluate and improve self-supervised learning models.
problem Self-supervised learning evaluation is limited to a single metric, providing little insight into model performance and improvement.
method Proposes an SSL risk decomposition that considers four error components: approximation, representation usability, probe generalization, and encoder generalization.
result Analysis of 169 SSL vision models reveals the main sources of error and provides insights for improving SSL models in specific settings.
New theory explains how self-supervised learning converges, advancing AI research.
problem Lack of precise theoretical explanation for self-supervised learning convergence.
method Synthesized Identifiability Theory with empirical evidence to propose Singular Identifiability Theory (SITh).
result SITh provides deeper insights into SSL's implicit data assumptions and advances representation learning.
A new SSL method improves medical image classification using global latent mixing.
problem Costly annotation of large-scale medical image data sets.
method Linear mixing of labeled and unlabeled data in both input and latent space.
result Improved performance in semi-supervised classification of thoracic disease and skin lesion.
Maximizes image representation dependence for self-supervised learning.
problem Learning meaningful image representations from unlabeled data.
method Maximizes Hilbert-Schmidt Independence Criterion (HSIC) between image transformations and identity.
result Matches state-of-the-art performance on ImageNet and other vision tasks.
Generative model creates synthetic unlabeled data for SSL.
problem Training SSL models without real unlabeled datasets.
method Meta-optimized synthetic samples generated from generative models.
result Synthetic samples improve SSL performance more efficiently than real unlabeled data.
DoubleMatch combines pseudo-labeling with self-supervision for SSL.
problem Lack of effective use of unlabeled data in SSL.
method Combines pseudo-labeling with self-supervised loss.
result Achieves state-of-the-art accuracies on multiple datasets.
Semi-supervised learning debiased for better performance.
problem Bias in SSL methods impairs theoretical guarantees and empirical performance.
method Simple debiasing approach to remove risk estimate bias.
result Debiased SSL methods provide better calibrated models.
Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that th…
Semi-Supervised Learning (SSL) algorithms have shown great potential in training regimes when access to labeled data is scarce but access to unlabeled data is plentiful. However, our experiments illustrate several shortcomings that prior SSL algorithms suffer from. In particular, poor performance when unlabeled and lab…
Solves biased pseudo-labels in imbalanced SSL by refining them.
problem Imbalanced class distributions in semi-supervised learning lead to biased pseudo-labels.
method Formulates a convex optimization problem to refine pseudo-labels and develops an efficient algorithm, DARP.
result Demonstrates the effectiveness of DARP in various imbalanced semi-supervised scenarios.
Semi-Supervised Learning (SSL) has achieved great success in overcoming the difficulties of labeling and making full use of unlabeled data. However, SSL has a limited assumption that the numbers of samples in different classes are balanced, and many SSL algorithms show lower performance for the datasets with the imbala…
Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.
problem Downstream task performances of GNN SSL on RDBs are poor.
method Proposed InfoNode to maximize mutual information between initial and final node representations.
result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.
As a newly emerging unsupervised learning paradigm, self-supervised learning (SSL) recently gained widespread attention, which usually introduces a pretext task without manual annotation of data. With its help, SSL effectively learns the feature representation beneficial for downstream tasks. Thus the pretext task play…
This work defines idealized SSL representations and improves existing methods.
problem Unclear characteristics of SSL representations leading to high downstream accuracies.
method Characterized ideal properties and derived necessary and sufficient conditions.
result Improved SSL methods and derived new objectives for contrastive and non-contrastive learning.
MTSSL optimizes threshold τ for better semi-supervised learning performance.
problem Optimizing the threshold τ for effective semi-supervised learning.
method Meta-Thresholding approach to optimize τ during training.
result Optimal values of τ are not necessary for achieving similar performance.
MET learns tabular data representations without data augmentations.
problem Lack of effective self-supervised learning methods for tabular data.
method Reconstruction-based approach using masked encoding, with separate representations for each coordinate and adversarial reconstruction loss.
result MET achieves state-of-the-art performance on five diverse tabular datasets, improving up to 9% over current methods.
SSL theory improves representation learning from raw data.
problem Challenges in SSL, including instability and collapse.
method Precise analysis of generalization performance with a theory-friendly setup.
result Insights for SSL practitioners on data augmentation, network architecture, and training algorithm.
Computer vision SSL methods show effectiveness on time series data.
problem Evaluate if computer vision SSL frameworks are effective on time series data.
method Evaluated on UCR and UEA archives, proposed a new method improving VICReg.
result Computer vision SSL frameworks can be effective on time series data.
A new probabilistic model for semi-supervised learning unifies various methods.
problem Combining different aspects of data distribution for semi-supervised learning.
method A probabilistic model that interprets and improves upon existing SSL methods.
result The model unifies various SSL methods and extends to neuro-symbolic learning.
Semi-supervised learning improves classification in high dimensions.
problem Combining labeled and unlabeled data for high-dimensional classification.
method Information theoretic and computational lower bounds analysis for feature selection.
result Semi-supervised learning is advantageous for classification in high dimensions.
This paper explores SSL for graph neural networks, improving performance on real-world datasets.
problem Leveraging unlabeled data for graph neural networks to improve deep learning performance.
method Empirical study of various SSL pretext tasks on graphs and proposing a new approach called SelfTask.
result Proposes SelfTask, achieving state-of-the-art performance on real-world datasets.
The study analyzes consistency-based SSL methods and proposes a new framework.
problem Improving performance in semi-supervised learning with minimal labeled data.
method Analysis of consistency-based SSL methods, including the Π-model, and proposing a new framework. result A simple extension of the Hidden Manifold Model that incorporates data-augmentation schemes.
From a machine learning perspective, the human ability localize sounds can be modeled as a non-parametric and non-linear regression problem between binaural spectral features of sound received at the ears (input) and their sound-source directions (output). The input features can be summarized in terms of the individual…
SSL framework identifies non-linear systems without labeled data.
problem System identification in non-linear environments without labeled data.
method Dynamics contrastive learning framework.
result SSL can identify non-linear dynamics in latent space.
Meta-Semi learns to optimize SSL with minimal hyper-parameter tuning.
problem Limited labeled data in SSL makes it impractical to tune many hyper-parameters.
method Meta-learning approach that dynamically reweights unlabeled data loss.
result Meta-Semi achieves competitive performance on various SSL tasks.
GAN-based semi-supervised learning improves classifier generalization.
problem Improving classifier performance with limited labeled data.
method Theoretical analysis of GAN-SSL, proving equivalence of discriminator optimization and supervised learning, and exploring conditions for perfect discriminator.
result GAN-SSL theoretically outputs a perfect discriminator on both labeled and unlabeled data.
Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, many of them have thus far been either inflexible, inefficient or …
Unified framework interprets SSL models, revealing biases.
problem Interpreting black-box SSL models for transparency.
method KREPES framework using Representer Landmarks and Nyström approximation.
result Direct audit of SSL latent space without supervision.
As a promising area in artificial intelligence, a new learning paradigm, called Small Sample Learning (SSL), has been attracting prominent research attention in the recent years. In this paper, we aim to present a survey to comprehensively introduce the current techniques proposed on this topic. Specifically, current S…
Active graph-based semi-supervised learning (AG-SSL) aims to select a small set of labeled examples and utilize their graph-based relation to other unlabeled examples to aid in machine learning tasks. It is also closely related to the sampling theory in graph signal processing. In this paper, we revisit the original fo…
Self-supervised learning performs better than supervised learning on imbalanced datasets.
problem The performance gap between balanced and imbalanced pre-training with self-supervised learning is smaller than with supervised learning.
method Systematic investigation of self-supervised learning under dataset imbalance, including experiments and theoretical analyses.
result Self-supervised representations are more robust to class imbalance than supervised representations.
MaxMatch improves SSL with worst-case consistency for better generalization.
problem Efficiently supervised learning with unlabeled data.
method Worst-case consistency regularization for SSL, providing a bound and an algorithm.
result The proposed method converges to a stationary point and improves generalization.
Enhances SSL methods with depth cues for better image understanding.
problem Lack of depth cues in 2D image pixel maps limits SSL performance.
method Integrates depth signals from a pretrained monocular RGB-to-depth model into contrastive learning frameworks.
result Improves SSL methods' robustness and generalization with depth signals.
RegMixMatch optimizes Mixup for semi-supervised learning by integrating high- and low-confidence samples.
problem Mixup degrades SSL performance by compromising artificial labels purity.
method RegMixMatch integrates high- and low-confidence samples, uses class-aware Mixup, and mitigates confirmation bias.
result RegMixMatch achieves state-of-the-art performance in SSL benchmarks.
New theory explains how equivariant self-supervised learning improves feature extraction.
problem Contrastive learning sacrifices useful features due to invariance to data augmentations.
method Information-theoretic perspective to understand E-SSL's generalization ability.
result Equivariant self-supervised learning creates synergy between equivariant and classification tasks.
Paper analyzes high-dimensional GMM for SSL, showing RMLE near-optimal performance.
problem Understanding when and why SSL works well with unlabeled data.
method Message-passing and state evolution methods for high-dimensional settings.
result RMLE achieves near-optimal performance in estimation and prediction with appropriate regularization.
Proposes SSL method for non-randomly sampled data.
problem Evaluation of prediction rules under non-random sampling.
method Two-step procedure with imputation and augmentation.
result Proposed method outperforms supervised methods in efficiency.
While the success of semi-supervised learning (SSL) is still not fully understood, Schölkopf et al. (2012) have established a link to the principle of independent causal mechanisms. They conclude that SSL should be impossible when predicting a target variable from its causes, but possible when predicting it from its ef…
A new method MixGDA combines mixup and gradient-based data augmentation for SSL.
problem Improving semi-supervised learning performance with limited labeled data.
method Gradient-based Data Augmentation (GDA) combined with mixup methods.
result MixGDA achieves state-of-the-art performance in various SSL benchmarks.
Unified framework for self-supervised learning via latent distribution matching.
problem Lack of a unifying theoretical framework for diverse SSL methods.
method Casting SSL as latent distribution matching (LDM): maximizing alignment and uniformity.
result Derives a Bayesian filtering model and proves identifiable latent representations.
A new SSL method uses instance-dependent thresholds to improve accuracy.
problem Improving semi-supervised learning by better selecting confident unlabeled instances.
method Proposes instance-dependent thresholds that vary based on the ambiguity and error rates of pseudo-labels for each unlabeled instance.
result Demonstrates that instance-dependent thresholds provide a probabilistic guarantee for correct pseudo-labels.
JRFs improve semi-supervised learning by balancing generation and classification.
problem Mode missing and mode covering issues in GANs and VAEs, and conflict between good classification and generation.
method Joint-stochastic-approximation random fields (JRFs) for deep undirected generative models.
result JRFs achieve good classification and generation results in SSL.
AUC-spec optimizes graph-based SSL for complex label distributions.
problem Training accurate models with scarce labeled data and abundant unlabeled data.
method Computes a low-dimensional representation that maximizes class separation via AUC optimization.
result AUC-spec achieves competitive results on synthetic and real-world datasets.
Semi-supervised learning improves prediction using unlabeled data.
problem Improving prediction performance using unlabeled data.
method General methodology for semi-supervised Empirical Risk Minimization (ERM) focusing on generalized linear regression.
result Adaptive SSL can achieve substantial improvement over supervised and null models in various settings.