Self-training improves model accuracy by refining pseudo-labels.
problem Improving semi-supervised learning with self-training.
method Theoretical insights into self-training algorithm with a focus on linear classifiers.
result Self-training iterations can improve model accuracy even if stuck in sub-optimal fixed points.
Since the creation of Generative Adversarial Networks (GANs), much work has been done to improve their training stability, their generated image quality, their range of application but nearly none of them explored their self-training potential. Self-training has been used before the advent of deep learning in order to …
Self-PU combines self-training with PU learning for improved binary classification.
problem Binary classification from unlabeled data and few labeled positives.
method Integrates self-paced training, self-calibrated loss, and self-distillation.
result Significantly improved performance on Alzheimer's Disease classification.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
Self-training improves neural sequence generation by correcting incorrect predictions.
problem Improving neural sequence generation models using unlabeled data.
method Injecting pseudo-parallel data (model predictions) into the labeled dataset and using dropout as a regularizer.
result Noisy self-training significantly improves performance on machine translation and text summarization benchmarks.
Doubly robust self-training improves semi-supervised learning by balancing labeled and pseudo-labeled data.
problem Improving semi-supervised learning performance with limited labeled data.
method Introduces doubly robust self-training, a method that combines labeled and pseudo-labeled data to balance between labeled-only and pseudo-labeled-only training.
result Demonstrates superior performance of doubly robust self-training on ImageNet and nuScenes datasets.
Paper stabilizes generative model training with synthetic data.
problem Self-consuming loops in generative model training.
method Introducing an idealized correction function and self-correction functions.
result Self-consuming loops can be exponentially more stable with the right correction.
Self-augmentation improves deep networks for few-shot learning with minimal training data.
problem Improving deep networks' generalization to unseen classes with limited training examples.
method Self-augmentation using self-mix and self-distillation techniques, combined with regional dropout and local representation learning.
result The method outperforms state-of-the-art few-shot learning methods on prevalent benchmarks.
Unified analysis of self-training for deep networks on unlabeled data.
problem Theoretical understanding of self-training for deep networks on unlabeled data.
method Unified theoretical analysis using expansion assumption and input-consistency regularization.
result Proves high accuracy of minimizers of population objectives based on self-training and input-consistency regularization.
Self-supervision improves GCNs' generalizability and robustness.
problem Improving graph convolutional networks' performance.
method Three mechanisms of self-supervision, multi-task learning, and graph adversarial training.
result Self-supervision enhances GCNs' robustness and generalizability.
Reinforcement learning improves self training for medical image segmentation.
problem Lack of labeled data in medical imaging.
method Integrating reinforcement learning into self training for complex segmentation networks.
result Improved segmentation performance with less labeled data.
Self-training improves gradual domain adaptation with unlabeled data.
problem Improving machine learning models' adaptability to gradually shifting data distributions.
method Proved upper bounds on self-training error, highlighted the importance of regularization and label sharpening, and demonstrated algorithmic insights.
result Self-training works well for gradual shifts, especially with small Wasserstein-infinity distance.
Self-adaptive training improves deep learning robustness.
problem Improving deep learning performance on corrupted data.
method Dynamic correction of problematic labels using model predictions.
result Self-adaptive training significantly improves generalization over ERM under various levels of noise.
Survey on self-supervised pre-training for neural networks using unlabeled data.
problem Improving model performance using unlabeled data.
method Pre-training on unlabeled data followed by task-specific adaptation.
result Enhanced model performance through self-supervised pre-training.
Self-supervised GAN prevents forgetting in sequential tasks.
problem Discriminator forgetting in GANs leads to training instability.
method Add self-supervision to the discriminator to maintain useful representations.
result Self-supervised GAN outperforms regular GANs in learning better representations.
Proposes a novel graph self-training method with EM regularization for semi-supervised node classification.
problem Handles noisy graph structures and feature spaces in semi-supervised node classification.
method Introduces an Expectation-Maximization (EM) regularization scheme for uncertainty-aware pseudo-label generation and model retraining.
result Significantly outperforms strong baselines by up to 2.5% in accuracy.
S4 learns new self-supervision automatically, improving accuracy with less human effort.
problem Lack of direct supervision in machine learning.
method Combines deep learning and probabilistic logic to automatically generate and verify new self-supervision.
result S4 can automatically propose accurate self-supervision, matching supervised methods with less human effort.
Self-training with noisy student-teacher boosts keyword spotting accuracy.
problem Robust keyword spotting in challenging conditions.
method Aggressive data augmentation and self-training with noisy student-teacher approach.
result Significant accuracy improvement in difficult conditions, up to 60%.
GUST framework improves self-training by estimating node uncertainty and generating pseudo-labels.
problem Over-confidence in pseudo-labels during self-training.
method Graph-based uncertainty-aware self-training with stochastic node labeling.
result GUST achieves state-of-the-art performance, especially in sparse labeled data settings.
Improved self-supervised learning for document images.
problem Performance of self-supervised pre-training on document images is poor.
method Proposed context-aware alternatives and a novel multi-modal method.
result Novel method outperforms other self-supervised methods on document image classification.
Paper proposes M3S training for GCNs on graphs with few labels.
problem Learning graph embeddings with few labeled nodes is challenging.
method Multi-Stage Self-Supervised (M3S) Training Algorithm combining self-supervised learning.
result M3S Training Algorithm improves GCNs' generalization on graphs with few labeled nodes.
Combining self-training and contrastive learning improves performance under distribution shift.
problem Improving performance under distribution shift using unlabeled data.
method Combining self-training and contrastive learning techniques.
result Combined method achieves 3-8% higher accuracy than either approach independently.
Self-supervised learning improves few-shot classification and segmentation on point clouds.
problem Efficiently learn from limited labeled data in point cloud applications.
method Hierarchical cover-tree partitioning for self-supervised pre-training; restricted to support set for few-shot learning.
result Self-supervised learning significantly improves downstream classification and segmentation accuracy.
LST improves few-shot classification by leveraging unlabeled data and meta-learning.
problem Challenges of few-shot classification due to limited labeled data.
method Semi-supervised meta-learning method (LST) that uses unlabeled data and a soft weighting network (SWN).
result Significant improvements over state-of-the-art methods on ImageNet benchmarks.
Self-supervised skip-tree training improves mathematical reasoning in language models.
problem Improving logical reasoning in language models for formal mathematics.
method Self-supervised language modeling on mathematical formulas, skip-tree task.
result Models trained on skip-tree task outperform standard models in mathematical reasoning tasks.
Boosts few-shot learning with self-supervised representations.
problem Improving generalization on small labeled datasets.
method Introduces self-supervised tasks as auxiliary loss functions.
result Reduces relative error rate by 5-25% on few-shot learning benchmarks.
This paper improves self-play learning in games by manipulating experience distributions.
problem Improving self-play learning in games through better experience sampling.
method Three approaches: weighted sampling, Prioritized Experience Replay, and diversifying trajectories.
result Major improvements in early training performance in some games, minor improvements overall.
RMT reveals self-regularization in neural networks, including traditional and heavy-tailed forms.
problem Understanding and quantifying self-regularization in neural networks.
method Application of Random Matrix Theory to analyze weight matrices of various neural network models.
result Identification of 5+1 phases of training in neural networks, including traditional and heavy-tailed self-regularization.
The paper investigates how class mean vectors enhance neural network classification.
problem Improving neural network performance in classification tasks.
method Exploring the role of class mean vectors in neural networks, including direct computation of weights, performance monitoring, and self-training.
result Empirical evidence suggests that using class mean vectors can significantly improve neural network performance on classification tasks.
SELF filters noisy labels to improve deep learning performance.
problem Overfitting to noisy labels in deep learning.
method Self-ensemble label filtering (SELF) using running averages of predictions.
result SELF improves task performance by filtering noisy labels dynamically.
Researchers develop PAIN to improve self-driving safety through adversarial training.
problem Overfitting and poor generalizability of neural networks in self-driving vehicles.
method PAIN combines adversarial training in CARLA simulation to generate edge cases.
result Trained self-driving vehicles are more resilient to environmental uncertainty and less prone to collisions.
The paper improves semi-supervised learning using f-divergences and α-Rényi divergences.
problem Improving semi-supervised learning with noisy pseudo-labels.
method Inspired by f-divergences and α-Rényi divergences, the paper develops new empirical risk functions and regularization techniques. result The new methods show better performance than traditional self-training methods, especially in noisy pseudo-label scenarios.
A study on optimizing self-attention in tabular data using Optimal Transport.
problem Improving efficiency and accuracy of self-attention in tabular classification tasks.
method Developed an OT-based algorithm to generate class-specific dummy Gaussian distributions and train an MLP.
result Achieved comparable accuracy to Transformers with reduced computational cost and efficiency.
Improved self-supervised denoising for Poisson-Gaussian noise.
problem Handling Poisson-Gaussian noise in self-supervised denoising.
method Extended blindspot model, improved training scheme without hyperparameters.
result Improved denoising performance on microscope image benchmarks.
Self-training in linear models shows a U-shaped test-risk curve due to signal forgetting and denoising.
problem Understanding the dynamics of iterative self-training in high-dimensional linear regression.
method Derivation of deterministic-equivalent recursions for prediction risk and effective noise, analysis of signal forgetting and denoising effects.
result An optimal early-stopping time is determined, and a U-shaped test-risk curve is observed.
Proposes a self-training method for automatic data annotation.
problem Lack of large annotated datasets for supervised deep learning.
method Deep Bayesian Self-Training using variational inference and clustering.
result Superior performance over standard Self-Training baselines.
Improves model performance with unlabeled test data.
problem Improving model robustness to distribution shifts.
method Self-supervised learning using a single unlabeled test sample.
result Improves performance on diverse image classification benchmarks.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that t…
Self-training improves weak classifiers in mixture models.
problem Improving weak classifiers in mixture models.
method Iterative self-training algorithm using pseudolabels and unlabeled data.
result Self-training converts weak learners to strong learners in mixture models.
Deep nearest neighbors outperform self-supervised methods in anomaly detection.
problem Anomaly detection using self-supervised deep methods.
method Simple nearest-neighbor approach on Imagenet pretrained features.
result Nearest-neighbor method outperforms self-supervised methods in accuracy, few shot generalization, training time, and noise robustness.
EBMs trained with ML are shown to behave like GANs with a self-adversarial loss.
problem Training EBMs with ML is intractable due to intractable unnormalized distributions.
method Replaced MCMC with deterministic gradient descent ODE solutions to study density induced by dynamics.
result EBM training is effectively a self-adversarial procedure rather than ML estimation.
Self-supervised method predicts clean signal and noise distribution from noisy images.
problem Blind denoising and noise estimation in biomedical images with limited clean data.
method Two neural networks jointly predict clean signal and noise distribution from noisy observations.
result Significantly outperforms state-of-the-art algorithms on six biomedical image datasets.
Self-distillation improves model performance but can lead to underfitting.
problem Understanding why self-distillation improves model performance and its limitations.
method Theoretical analysis of self-distillation in Hilbert space with ℓ2 regularization. result Self-distillation modifies regularization by limiting the number of basis functions, potentially leading to underfitting.
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. RDIS fills missing values in time series data explicitly.
problem Missing values in time series data.
method Random Drop Imputation with Self-training.
result RDIS achieves competitive results on real-world datasets.
Training-free method improves text-to-image generation quality.
problem Error accumulation in simultaneous token updates of masked diffusion models.
method Training-free self-correction framework exploiting inductive biases.
result Significantly improved generation quality on text-to-image tasks.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.