New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
Novel bio-inspired masking for robust speech emotion recognition.
problem Noise degradation in speech emotion recognition.
method Cochlear cepstrogram-based contrastive learning with temporal and frequency masking.
result Improved speech emotion recognition performance on K-EmoCon benchmark.
AaSP improves audio self-supervised learning by addressing aliasing issues.
problem Alias issues in audio spectrogram transformers.
method AaSP combines aliasing-aware patch representation, teacher-student masked modeling, cross-attention predictor, and contrastive regularization.
result AaSP learns more stable representations that integrate high-frequency cues.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
New lens on self-supervised learning: parameter identifiability.
problem Identifying good features for downstream tasks in self-supervised learning.
method Consider data from a parametric probabilistic model, train a predictor, and ask if ground truth parameters can be read off the optimal predictor.
result Some prediction tasks yield identifiability, others do not, revealing a rich landscape.
SiamJEPA uses Siamese student encoders to improve JEPA-based representation learning.
problem Improving self-supervised representation learning in JEPA models.
method Proposes SiamJEPA with masked Siamese student encoders and EMA teacher network.
result Siamese student encoders improve representation separability and learning speed.
T-JEPA learns tabular data representations without augmentations, outperforming traditional methods.
problem Challenges in self-supervised learning for tabular data due to lack of data augmentations.
method T-JEPA uses a Joint Embedding Predictive Architecture (JEPA) to predict latent representations of different subsets of features within the same sample.
result Significant improvement in classification and regression tasks, outperforming traditional methods.
Unified theory for representation learning using learnable functions.
problem Insufficient theoretical understanding of unsupervised and self-supervised learning.
method Discriminative theoretical framework for analyzing sample complexity.
result Learnable regularization functions can reduce the amount of labeled data needed.
MET learns tabular data representations without data augmentations.
problem Lack of effective self-supervised learning methods for tabular data.
method Reconstruction-based approach using masked encoding, with separate representations for each coordinate and adversarial reconstruction loss.
result MET achieves state-of-the-art performance on five diverse tabular datasets, improving up to 9% over current methods.
ViCE uses superpixels to enhance self-supervised learning for better dense visual embeddings.
problem Lack of high-resolution feature maps from self-supervised models.
method Superpixels for dense representation learning, contrasting over regions.
result Improves unsupervised semantic segmentation on benchmarks like Cityscapes and COCO.
Theoretical analysis of vision transformers' performance with MAE and CL objectives.
problem Understanding the distinct representations learned by vision transformers with MAE and CL objectives.
method Modeling visual data distribution and analyzing ViTs training dynamics with gradient descent.
result ViTs trained with MAE objectives learn both global and local features, while CL-trained ViTs favor global features.
We consider the effect of structure-agnostic and structure-dependent masking schemes when training a universal marginaliser (arXiv:1711.00695) in order to learn conditional distributions of the form P(xi∣xb), where xi is a given random variable and xb is some arbitrary sub…
This paper improves anomaly detection in lane rendering images for safer navigation.
problem Anomalies in lane rendering images can mislead drivers, posing safety risks.
method Proposes a four-phase pipeline using Transformer models, self-supervised pre-training, and fine-tuning.
result The pipeline enhances detection accuracy and reduces training time.
A new method detects anomalous sounds using self-supervised learning.
problem Detecting anomalous sounds in noisy data.
method Flow-based self-supervised density estimation using Normalizing Flows.
result The method improves anomaly detection performance significantly.
Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and …
Proposes a new approach to time series representation learning by embedding patches independently.
problem Capturing dependencies between time series patches is not optimal for representation learning.
method 1) Patch reconstruction task, 2) Patch-wise MLP, 3) Complementary contrastive learning.
result Improves time series forecasting and classification performance compared to state-of-the-art models.
This work analyzes the role of data augmentation in self-supervised learning using RKHS approximation and regression.
problem Limited theoretical understanding of the role of data augmentation in self-supervised learning.
method Geometric characterization of the target function given by augmentation, proving generalization bounds.
result Two generalization bounds are derived, one free of model complexity, the other specific to near-optimal encoders.
RELAX provides first attribution-based explanations for representations.
problem Lack of methods to explain what influences learned representations.
method RELAX, a first approach for attribution-based explanations of representations, measuring similarities in representation space.
result Significantly outperforms gradient-based baseline and models uncertainty in explanations.
Transformer model removes noise from light curves efficiently.
problem Challenges in processing astrophysical light curves due to noise.
method Denoising Time Series Transformer (DTST) model trained with masked objective.
result DTST model excels at removing noise and outliers in time series datasets.
Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
daep learns from irregular, multimodal astronomical data.
problem Learning from irregular, multimodal astronomical sequences.
method Diffusion Autoencoder with Perceivers (daep) tokenizes, compresses, and reconstructs data.
result daep outperforms VAE and maep baselines in reconstruction and fine-scale structure preservation.
This study predicts parking availability using multi-source data and a self-supervised learning enhanced transformer.
problem Accurate parking availability prediction to support urban planning and management.
method Proposes SST-iTransformer, a self-supervised learning enhanced spatio-temporal inverted transformer, integrating multi-source data.
result SST-iTransformer achieves state-of-the-art performance in parking availability prediction.
SLiCE learns contextual node embeddings for link prediction in heterogeneous networks.
problem Link prediction requires specific contextual information not captured by static node embeddings.
method Self-supervised pre-training with localized attention mechanisms.
result SLiCE significantly outperforms existing methods on link prediction tasks.
We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-o…
Recent state-of-the-art image segmentation algorithms are mostly based on deep neural networks, thanks to their high performance and fast computation time. However, these methods are usually trained in a supervised manner, which requires large number of high quality ground-truth segmentation masks. On the other hand, c…
EarnMore uses masked stock representations to train RL agents for customizable stock pools efficiently.
problem Training RL agents for customizable stock pools (CSPs) is computationally expensive and unstable.
method EarnMore introduces a mechanism to mask out stocks outside the target pool, learns meaningful stock representations, and uses a re-weighting mechanism to focus on favorable stocks.
result EarnMore significantly outperforms state-of-the-art baselines in profit metrics with over 40% improvement.
S4 learns new self-supervision automatically, improving accuracy with less human effort.
problem Lack of direct supervision in machine learning.
method Combines deep learning and probabilistic logic to automatically generate and verify new self-supervision.
result S4 can automatically propose accurate self-supervision, matching supervised methods with less human effort.
This research shows unsupervised GANs can perform object segmentation without labels.
problem Performing object segmentation without pixel or image-level labels.
method Used large-scale unsupervised GAN models to differentiate foreground from background.
result Demonstrated high-quality saliency masks and new state-of-the-art performance.
Self-supervised learning excels in topic modeling by being less model-specific.
problem How self-supervised learning discovers useful representations in topic models.
method Applying self-supervised learning objectives to topic model-generated data.
result Self-supervised learning objectives can recover useful posterior information for topic models, outperforming misspecified models.
ST-MTM models complex time series by decomposing and masking seasonal and trend components.
problem Forecasting complex time series with intricate temporal variations.
method Seasonal-Trend Decomposition with Masking and Contrastive Learning.
result ST-MTM achieves superior forecasting performance compared to existing methods.
Survey of self-supervised learning methods in computer vision, NLP, and graph learning.
problem Manual labeling and vulnerability to attacks in supervised learning.
method Generative, contrastive, and generative-contrastive approaches.
result Self-supervised learning achieves high performance in representation learning.
MaskCycleGAN-VC improves voice conversion without parallel data.
problem Limited ability to convert mel-spectrogram data without parallel data.
method Integrates a novel auxiliary task called filling in frames (FIF) to learn time-frequency structures.
result MaskCycleGAN-VC outperforms existing methods with similar model size.
Self-supervision improves GCNs' generalizability and robustness.
problem Improving graph convolutional networks' performance.
method Three mechanisms of self-supervision, multi-task learning, and graph adversarial training.
result Self-supervision enhances GCNs' robustness and generalizability.
This paper analyzes self-supervised learning from a multi-view perspective.
problem Understanding and optimizing self-supervised learning from multi-view data.
method Information-theoretical framework to understand and design self-supervised learning objectives.
result Self-supervised representations can extract task-relevant information and discard task-irrelevant information.
Generative model explains self-supervised learning across various tasks.
problem Lack of theoretical understanding of self-supervised learning methods.
method Generative latent variable model for self-supervised learning.
result Improves representation learning performance and narrows the gap between generative and discriminative methods.
Novel approach to contextual bandits using self-supervised learning.
problem Exploiting rich data representations in computer vision without explicit labels.
method Combining contextual bandit objective with self-supervision objective.
result Substantial gains in cumulative reward on computer vision datasets.
This work improves adaptive conformal prediction using self-supervised learning.
problem Improving the adaptability of conformal prediction intervals.
method Train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
result Empirically demonstrates the benefit of additional information in improving the efficiency (width), deficit, and excess of conformal prediction intervals.
Improved self-supervised learning for document images.
problem Performance of self-supervised pre-training on document images is poor.
method Proposed context-aware alternatives and a novel multi-modal method.
result Novel method outperforms other self-supervised methods on document image classification.
Paper tackles self-supervised learning for non-homophilous graphs.
problem Existing self-supervised learning methods assume homophilous graphs, but real-world graphs often lack this assumption.
method Develops a decoupled self-supervised learning (DSSL) framework that decouples different semantics between neighborhoods.
result DSSL framework achieves better performance on various graph benchmarks compared to competitive baselines.
We present a technique to improve the transferability of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. While recent approaches for self-supervised learning have shown the benefits of training on large unlabeled datasets, we find improvements in …
Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a varie…
Improved self-supervised learning on ImageNet achieves top-1 accuracy of 77.1%.
problem Self-supervised ResNets underperform supervised learning on ImageNet.
method ReLICv2 combines explicit invariance loss with contrastive objective over varied data views.
result ReLICv2 achieves 77.1% top-1 accuracy on ImageNet, improving over previous state-of-the-art by 1.5%.
DM improves self-supervised transfer learning by matching target distributions.
problem Improving self-supervised transfer learning performance.
method Distribution Matching (DM) method that drives representation distribution towards a predefined reference distribution.
result DM outperforms existing methods on target classification tasks.
SASSL improves self-supervised learning by preserving image structure.
problem Distorted augmented samples in self-supervised learning.
method Neural Style Transfer to decouple semantic and stylistic attributes.
result Boosts ImageNet top-1 accuracy by up to 2 percentage points.
MARGE learns to reconstruct text by paraphrasing, achieving strong performance across multiple tasks.
problem Training sequence-to-sequence models with limited supervision.
method Unsupervised multi-lingual multi-document paraphrasing objective.
result Strong zero-shot performance on document translation and various tasks in multiple languages.
Survey on self-supervised pre-training for neural networks using unlabeled data.
problem Improving model performance using unlabeled data.
method Pre-training on unlabeled data followed by task-specific adaptation.
result Enhanced model performance through self-supervised pre-training.
The paper investigates the effects of invalid action masking in policy gradient algorithms.
problem Invalid actions in policy gradient algorithms can lead to suboptimal performance.
method The paper provides theoretical justification and empirical demonstrations of the importance of invalid action masking.
result Invalid action masking is crucial as the number of invalid actions increases.