Self-supervision improves GCNs' generalizability and robustness.
problem Improving graph convolutional networks' performance.
method Three mechanisms of self-supervision, multi-task learning, and graph adversarial training.
result Self-supervision enhances GCNs' robustness and generalizability.
S4 learns new self-supervision automatically, improving accuracy with less human effort.
problem Lack of direct supervision in machine learning.
method Combines deep learning and probabilistic logic to automatically generate and verify new self-supervision.
result S4 can automatically propose accurate self-supervision, matching supervised methods with less human effort.
Improved self-supervised learning for document images.
problem Performance of self-supervised pre-training on document images is poor.
method Proposed context-aware alternatives and a novel multi-modal method.
result Novel method outperforms other self-supervised methods on document image classification.
Survey on self-supervised pre-training for neural networks using unlabeled data.
problem Improving model performance using unlabeled data.
method Pre-training on unlabeled data followed by task-specific adaptation.
result Enhanced model performance through self-supervised pre-training.
We present a technique to improve the transferability of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. While recent approaches for self-supervised learning have shown the benefits of training on large unlabeled datasets, we find improvements in …
Self-supervised skip-tree training improves mathematical reasoning in language models.
problem Improving logical reasoning in language models for formal mathematics.
method Self-supervised language modeling on mathematical formulas, skip-tree task.
result Models trained on skip-tree task outperform standard models in mathematical reasoning tasks.
Improved self-supervised denoising for Poisson-Gaussian noise.
problem Handling Poisson-Gaussian noise in self-supervised denoising.
method Extended blindspot model, improved training scheme without hyperparameters.
result Improved denoising performance on microscope image benchmarks.
Deep nearest neighbors outperform self-supervised methods in anomaly detection.
problem Anomaly detection using self-supervised deep methods.
method Simple nearest-neighbor approach on Imagenet pretrained features.
result Nearest-neighbor method outperforms self-supervised methods in accuracy, few shot generalization, training time, and noise robustness.
Self-supervised learning improves few-shot classification and segmentation on point clouds.
problem Efficiently learn from limited labeled data in point cloud applications.
method Hierarchical cover-tree partitioning for self-supervised pre-training; restricted to support set for few-shot learning.
result Self-supervised learning significantly improves downstream classification and segmentation accuracy.
GANs involve training two networks in an adversarial game, where each network's task depends on its adversary. Recently, several works have framed GAN training as an online or continual learning problem. We focus on the discriminator, which must perform classification under an (adversarially) shifting data distribution…
Self-supervised method predicts clean signal and noise distribution from noisy images.
problem Blind denoising and noise estimation in biomedical images with limited clean data.
method Two neural networks jointly predict clean signal and noise distribution from noisy observations.
result Significantly outperforms state-of-the-art algorithms on six biomedical image datasets.
Graph Convolutional Networks(GCNs) play a crucial role in graph learning tasks, however, learning graph embedding with few supervised signals is still a difficult problem. In this paper, we propose a novel training algorithm for Graph Convolutional Network, called Multi-Stage Self-Supervised(M3S) Training Algorithm, co…
We describe a novel weakly labeled Audio Event Classification approach based on a self-supervised attention model. The weakly labeled framework is used to eliminate the need for expensive data labeling procedure and self-supervised attention is deployed to help a model distinguish between relevant and irrelevant parts …
Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a varie…
Self-supervised ECG learning improves emotion recognition.
problem Improving emotion recognition from ECG signals.
method Multi-task deep learning framework with signal transformations as pretext tasks.
result Significant performance improvement in emotion classification.
DoubleMatch combines pseudo-labeling with self-supervision for SSL.
problem Lack of effective use of unlabeled data in SSL.
method Combines pseudo-labeling with self-supervised loss.
result Achieves state-of-the-art accuracies on multiple datasets.
SSNAS finds neural architectures without labeled data.
problem Limited labeled data for NAS.
method Self-supervised learning for NAS.
result Comparable results to supervised NAS with labeled data.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
Self-supervised learning helps train deep features without needing lots of labeled data.
problem Annotation bottleneck in deep learning.
method Four main families of self-supervised approaches applied to various data modalities.
result Self-supervised methods can now rival fully supervised pre-training across multiple data types.
Simplified non-contrastive learning avoids representation collapse.
problem Training failure modes in self-supervised learning.
method Hyperdimensional computing and inductive bias.
result The approach avoids representation collapses.
While deep learning has been incredibly successful in modeling tasks with large, carefully curated labeled datasets, its application to problems with limited labeled data remains a challenge. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through a combin…
This work improves adaptive conformal prediction using self-supervised learning.
problem Improving the adaptability of conformal prediction intervals.
method Train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
result Empirically demonstrates the benefit of additional information in improving the efficiency (width), deficit, and excess of conformal prediction intervals.
The paper develops an asymptotic theory of self-supervised pre-training.
problem Sharpness of current rates in self-supervised pre-training and their accuracy.
method Two-stage M-estimation and tools from Riemannian geometry.
result Characterization of the limiting distribution of the downstream test risk.
Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and …
Self-supervised learning improves RUL prediction with limited data in fatigue damage prognosis.
problem Limited labelled data for RUL prediction in fatigue damage prognosis.
method Pre-training deep learning models on unlabelled sensor data using self-supervised learning.
result Self-supervised pre-trained models significantly outperform non-pre-trained models in RUL prediction with scarce labelled data.
This paper improves anomaly detection in lane rendering images for safer navigation.
problem Anomalies in lane rendering images can mislead drivers, posing safety risks.
method Proposes a four-phase pipeline using Transformer models, self-supervised pre-training, and fine-tuning.
result The pipeline enhances detection accuracy and reduces training time.
Self-supervised policy adapts after deployment without rewards.
problem Generalizing reinforcement learning policies across different environments.
method Uses self-supervision to train policies in new environments without reward signals.
result Significant improvements in generalization across diverse environments.
Self-supervised learning, which learns by constructing artificial labels given only the input signals, has recently gained considerable attention for learning representations with unlabeled datasets, i.e., learning without any human-annotated supervision. In this paper, we show that such a technique can be used to sign…
Paper tackles self-supervised learning for non-homophilous graphs.
problem Existing self-supervised learning methods assume homophilous graphs, but real-world graphs often lack this assumption.
method Develops a decoupled self-supervised learning (DSSL) framework that decouples different semantics between neighborhoods.
result DSSL framework achieves better performance on various graph benchmarks compared to competitive baselines.
AaSP improves audio self-supervised learning by addressing aliasing issues.
problem Alias issues in audio spectrogram transformers.
method AaSP combines aliasing-aware patch representation, teacher-student masked modeling, cross-attention predictor, and contrastive regularization.
result AaSP learns more stable representations that integrate high-frequency cues.
T-JEPA learns tabular data representations without augmentations, outperforming traditional methods.
problem Challenges in self-supervised learning for tabular data due to lack of data augmentations.
method T-JEPA uses a Joint Embedding Predictive Architecture (JEPA) to predict latent representations of different subsets of features within the same sample.
result Significant improvement in classification and regression tasks, outperforming traditional methods.
BetaDataWeighter learns weights for unlabelled data to improve self-supervised learning accuracy.
problem Improving unsupervised representations with domain shift between unlabelled and target data.
method Learning Bayesian instance weights for unlabelled data to prioritize useful instances.
result BetaDataWeighter achieves highest average accuracy and prunes up to 78% of images without significant loss in accuracy.
Novel approach to contextual bandits using self-supervised learning.
problem Exploiting rich data representations in computer vision without explicit labels.
method Combining contextual bandit objective with self-supervision objective.
result Substantial gains in cumulative reward on computer vision datasets.
Conditional GANs are at the forefront of natural image synthesis. The main drawback of such models is the necessity for labeled data. In this work we exploit two popular unsupervised learning techniques, adversarial training and self-supervision, and take a step towards bridging the gap between conditional and uncondit…
New method separates objects from images using deep neural networks trained to inpaint.
problem Fully self-supervised instance separation of occluded objects in images.
method Maximizes independence of two image regions given a fully self-supervised inpainting network.
result Method achieves similar segmentation performance to fully supervised methods on microscopy image datasets.
The paper accelerates LLM inference by adding early exit heads trained in a self-supervised manner.
problem Inference speed in large language models (LLMs) is slow and resource-intensive.
method Adding self-supervised early exit heads at intermediate transformer layers to stop computation early based on confidence thresholds.
result Entropy provides the most reliable confidence metric for stopping computation early.
Improved conversion rate prediction in online advertising using self-supervised pre-training.
problem Data sparsity and calibration issues in predicting conversions given clicks.
method Self-supervised pre-training on all conversion events to enrich CVR prediction model without compromising calibration.
result Improvements in offline training and online A/B tests, with full deployment to Yahoo native advertising system.
In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions. We turn a single unlabeled test sample into a self-supervised learning problem, on which we update the model parameters before making a predi…
New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
SidAE combines autoencoders and Siamese networks for self-supervised feature extraction.
problem Efficiently extract meaningful features from unlabeled data.
method Proposes SidAE, a combination of autoencoders and Siamese networks for self-supervised image classification.
result SidAE outperforms self-supervised baselines across multiple datasets and scenarios, especially with limited labeled data.
Plan2Explore learns new tasks efficiently through self-supervised planning.
problem Challenges in reinforcement learning, especially task-specific learning and sample efficiency.
method Self-supervised exploration and fast adaptation to new tasks through efficient planning.
result Plan2Explore outperforms prior methods in learning new tasks without supervision.
Paper analyzes self-supervised image denoising with denatured data.
problem Understanding the performance of self-supervised image denoising with denatured data.
method Theoretical analysis and numerical experiments on a denoising algorithm.
result Theoretical analysis shows the algorithm finds desired solutions to the optimization problem.
Self-supervised model detects phoneme boundaries without annotations.
problem Unsupervised phoneme segmentation without manual annotations.
method Convolutional neural network trained with Noise-Contrastive Estimation.
result Model outperforms baselines on TIMIT and Buckeye corpora.
This paper reviews self-supervised learning methods for time series anomaly detection.
problem Challenges in traditional unsupervised methods for time series anomaly detection.
method Self-supervised learning techniques for time series anomaly detection.
result Enhanced performance of anomaly detectors through self-supervised learning.
Self-supervised methods learn from noisy data alone, useful for imaging problems.
problem Inferring signals from noisy and incomplete observations.
method Learning a solver from measurement data alone, without ground-truth references.
result Self-supervised methods can learn meaningful estimates from noisy data.
New method learns physical meanings in learned representations for better downstream tasks.
problem Lack of explicit physical meanings in learned representations from self-supervised learning.
method Analysis-by-synthesis approach with coordinated sampling and training.
result Trained models can control articulatory synthesizers to speak like humans and generalize well.
New bound proves rationality helps generalization in self-supervised learning.
problem Proving generalization gap in self-supervised learning.
method Proving upper bound on generalization gap for classifiers using rationality and self-supervised representations.
result Generalization gap tends to zero if classifier complexity is low relative to number of samples.
Paper proposes a method to train robust neural networks without labeled data.
problem Training robust neural networks without class labels.
method Adversarial contrastive learning framework using unlabeled data.
result Robust Contrastive Learning (RoCL) achieves comparable robust accuracy to supervised methods and significantly improved robustness.