BYOL learns useful representations without batch statistics.
problem BYOL's reliance on batch statistics for representation learning.
method Training BYOL without batch normalization and using alternative normalization schemes.
result BYOL performance comparable to vanilla BYOL without batch normalization.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
BYOL learns image representations without negative pairs.
problem Self-supervised image representation learning.
method Two neural networks interact and learn from each other, updating the target network with a slow-moving average of the online network.
result Achieves state-of-the-art performance on ImageNet.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
Develops a new self-supervised learning method combining contrastive and non-contrastive approaches.
problem Leveraging unlabeled data for representation learning, especially with high variance and low batch sizes.
method Converts a contrastive method (Spectral Contrastive Loss) into a non-contrastive form (MINC loss) to reduce variance and mutual information.
result MINC loss consistently improves upon the Spectral Contrastive loss baseline in learning image representations.
BYOV combines SSL and Bayesian methods for uncertainty estimation.
problem Model uncertainty in applications.
method Combines Bootstrap Your Own Latent (BYOL) and Bayes by Backprop (BBB).
result BYOV improves model calibration and reliability with various augmentations.
Enhances SSL methods with depth cues for better image understanding.
problem Lack of depth cues in 2D image pixel maps limits SSL performance.
method Integrates depth signals from a pretrained monocular RGB-to-depth model into contrastive learning frameworks.
result Improves SSL methods' robustness and generalization with depth signals.
CLOCS uses contrastive learning to improve cardiac signal representations.
problem Lack of labelled data in cardiac signal analysis.
method Contrastive learning across space, time, and patients.
result CLOCS outperforms state-of-the-art methods and achieves strong generalization.
SASSL improves self-supervised learning by preserving image structure.
problem Distorted augmented samples in self-supervised learning.
method Neural Style Transfer to decouple semantic and stylistic attributes.
result Boosts ImageNet top-1 accuracy by up to 2 percentage points.
Unified framework interprets SSL models, revealing biases.
problem Interpreting black-box SSL models for transparency.
method KREPES framework using Representer Landmarks and Nyström approximation.
result Direct audit of SSL latent space without supervision.
New method proves non-contrastive self-supervised learning learns useful features.
problem Understanding how non-contrastive self-supervised learning (NS-SL) learns useful features.
method Proved in a linear network, NS-SL learns a desirable projection matrix and reduces sample complexity. Suggested weight decay acts as an implicit threshold.
result DirectCopy, a simpler and more efficient algorithm, outperforms DirectPred on various datasets.
Maximizes image representation dependence for self-supervised learning.
problem Learning meaningful image representations from unlabeled data.
method Maximizes Hilbert-Schmidt Independence Criterion (HSIC) between image transformations and identity.
result Matches state-of-the-art performance on ImageNet and other vision tasks.
Feature normalization prevents collapse in non-contrastive learning dynamics.
problem Non-contrastive learning can collapse into a single point due to lack of repulsive force.
method Extended previous theory based on L2 loss to cosine loss, considering feature normalization.
result Cosine loss induces stable equilibrium, preventing collapse even with insufficient repulsive force.
This work provides a scaling rule for model EMA optimization across batch sizes.
problem Training dynamics and performance differences across batch sizes when using model EMA.
method Developed a scaling rule for model EMA optimization, demonstrating its validity across various architectures and data modalities.
result Enabled SSL methods like BYOL to train at larger batch sizes without performance degradation.
This work explains how maximizing latent correlations across multiple data views helps in identifying shared and private components.
problem Understanding how to identify shared and private components in multiview data.
method An intuitive generative model of multiview data is adopted, and latent correlation maximization is shown to guarantee the extraction of shared components.
result Latent correlation maximization guarantees the extraction of shared components across views and disentangles private information.
New method improves unsupervised feature learning for natural data.
problem Natural data's correlated and long-tail distribution challenges instance-level contrastive learning.
method Cross-level instance-group discrimination (CLD) to integrate between-instance similarity.
result CLD achieves new state-of-the-art performance on various datasets.
New method removes contrastive loss by adding a prediction head, revealing learning mechanisms.
problem Understanding why neural networks learn competitive representations despite trivial optima.
method Empirical and theoretical analysis of a trainable, identity-initialized prediction head.
result The trainable prediction head enables learning all features, preventing dimensional collapse.