Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. Self-distillation improves model performance but can lead to underfitting.
problem Understanding why self-distillation improves model performance and its limitations.
method Theoretical analysis of self-distillation in Hilbert space with ℓ2 regularization. result Self-distillation modifies regularization by limiting the number of basis functions, potentially leading to underfitting.
Improved self-distillation reduces label noise and enhances model accuracy.
problem Label noise in multi-class classification.
method Label averaging and refined partial labels.
result Single-round self-distillation achieves comparable performance to multi-round distillation.
Self-distillation optimally improves model performance in spiked covariance models.
problem Improving model performance in spiked covariance models.
method Developed spectral shrinkage estimators and analyzed self-distillation.
result Self-distillation achieves optimal performance among spectral shrinkage estimators for spiked covariance matrices.
Improved generalization with iterative self-distillation using weighted ground-truth targets.
problem Improving generalization accuracy in neural networks.
method Iterative kernel regression with weighted ground-truth targets and ℓ2 regularization. result Closed-form solution for optimal weighting parameter and efficient estimation.
Optimal self-distillation improves generative models' velocity risk and mode recovery.
problem Improving generative models' velocity risk and mode recovery.
method Proved optimal self-distillation for rectified flow via linear probing, derived mixing coefficient, and provided validation tuning.
result Optimal self-distillation improves velocity risk and mode recovery.
Self distillation boosts CNN accuracy without increasing model size.
problem Improving CNN accuracy in resource-limited domains.
method Divide and compress knowledge within the network structure.
result Average accuracy improvement of 2.65% across various networks.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
New method regularizes deep networks by distilling self-knowledge.
problem Overfitting in deep neural networks.
method Self-knowledge distillation to regularize class-wise predictions.
result Significant improvement in generalization and calibration.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
Self-distillation improves constrained language generation by aligning models with target distributions.
problem Sparse and uninformative reward signals in constrained generation settings.
method Iteratively refining the base model through self-distillation, incorporating learned twist functions and proposals.
result Substantial gains in generation quality through improved model alignment with target distributions.
Two approaches extend knowledge distillation to Gaussian Processes, showing relationships to existing methods.
problem Applying knowledge distillation to Gaussian Processes for regression and classification.
method Data-centric and distribution-centric approaches to extend distillation to GPR and GPC.
result Distribution-centric approach for GPC approximately corresponds to data duplication and scaling.
Self-knowledge distillation improves NLP performance.
problem Efficient knowledge representation in NLP models.
method A new knowledge distillation method using soft target probabilities from the training model itself.
result Improves performance on language model and neural machine translation tasks.
SARD improves deep learning clinical prediction performance.
problem Deep learning models struggle to match linear models in healthcare predictions.
method Reverse Distillation to initialize deep models, combined with contextual and temporal embeddings.
result SARD outperforms state-of-the-art methods on clinical prediction outcomes.
This paper improves ASR performance by aligning frames more accurately.
problem Disagreement between teacher-student models in frame-level alignment.
method Introduces self-knowledge distillation (SKD) to guide frame-level alignment.
result Improves both resource efficiency and performance in ASR.
FAST-DAD distills complex ensemble models into faster, more accurate individual models.
problem Deploying complex AutoML ensemble predictors on tabular data is slow, large, and opaque.
method Data augmentation strategy based on Gibbs sampling from a self-attention pseudolikelihood estimator.
result FAST-DAD distillation produces significantly better individual models than standard training.
S2D efficiently trains models to estimate uncertainty without increasing resource costs.
problem Efficiently estimating uncertainty in deep learning models for safety-critical applications.
method Self-distribution distillation (S2D) approach to train a single model for uncertainty estimation.
result S2D models outperform standard models and Monte-Carlo dropout in uncertainty estimation.
Study shows optimal self-distillation improves model performance on noisy data.
problem Improving model performance on noisy Gaussian mixture data.
method Hyperparameter-tuned multi-stage self-distillation with a linear classifier, using replica method.
result Primary driver of SD's performance improvement is denoising through hard pseudo-labels.
SAIL improves graph node representation learning by distilling knowledge between graphs.
problem Improving graph node representation learning with GNNs in unsupervised scenarios.
method SAIL framework with intra- and inter-graph knowledge distillation.
result SAIL consistently outperforms state-of-the-art baselines on various benchmark datasets.
Self-Distilled Disentanglement improves counterfactual predictions by separating variables.
problem Improving counterfactual predictions in the presence of confounders and unobserved variables.
method Self-Distilled Disentanglement framework based on information theory.
result Effective counterfactual inference in synthetic and real-world datasets.
SFAVEL distills features from language models for fact verification without annotations.
problem Fact verification with semantically meaningful and compact features.
method Self-supervised pretraining using contrastive loss with language models.
result Achieved state-of-the-art results on FB15k-237 and FEVER.
Paper explains how early stopping helps distillation in overparameterized neural networks.
problem Understanding how overparameterized neural networks can improve with early stopping.
method Introducing Anisotropic Information Retrieval (AIR) to justify early stopping in distillation.
result Self-distillation algorithm improves over just early stopping, leading to better generalization.
A new framework for deep learning from multiple experts tackles long-tailed data issues.
problem Training deep networks on imbalanced data distributions.
method Learning From Multiple Experts (LFME) framework, involving self-paced expert selection and curriculum instance selection.
result LFME achieves superior performance compared to state-of-the-art methods.
Self-augmentation improves deep networks for few-shot learning with minimal training data.
problem Improving deep networks' generalization to unseen classes with limited training examples.
method Self-augmentation using self-mix and self-distillation techniques, combined with regional dropout and local representation learning.
result The method outperforms state-of-the-art few-shot learning methods on prevalent benchmarks.
Self-distillation improves model performance in noisy label settings.
problem Improving model accuracy in supervised learning with noisy labels.
method Analyzes self-distillation in two supervised learning problems with noisy labels, using theoretical and empirical approaches.
result Optimal self-distillation parameter is greater than 1 in high label noise regimes, outperforming traditional methods.
Proposes LsrKD and MrKD to improve neural network training performance.
problem Improving neural network training performance, especially on deep networks.
method Extends Label Smoothing Regularization with Self-Knowledge Distillation, introducing LsrKD and MrKD.
result LsrKD and MrKD significantly improve training performance on deep neural networks.
PS-KD distills a model's own knowledge to soften hard targets during training.
problem Improving generalization of deep neural networks by softening hard targets.
method Progressive self-knowledge distillation (PS-KD) that progressively distills a model's own knowledge to soften hard targets.
result PS-KD improves accuracy and provides high quality of confidence estimates in terms of calibration and ordinal ranking.
Faster WIND accelerates iterative BOND for LLM alignment.
problem Iterative BOND is inefficient in practice due to sample and computation inefficiency.
method Unified game-theoretic connection to self-play alignment, WIND framework with efficient algorithms.
result WIND variant achieves superior sample efficiency and faster computation.
To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To …
ProSMIN improves representation quality through probabilistic self-supervised learning.
problem Improving representation quality in self-supervised learning.
method ProSMIN uses two neural networks, online and target, to learn diverse representations through knowledge distillation and a modified scoring rule loss function.
result ProSMIN achieves superior accuracy and calibration on various downstream tasks.
This work compresses BERT into simple neural networks using unlabeled data.
problem Compression of large BERT models for practical use in downstream tasks.
method Leverage unlabeled transfer data to distill BERT into simple RNN models with hard and soft distillation.
result Simple RNN models can match or exceed BERT performance with up to 26x parameter compression.
A new method for distilling predictions from a teacher model to a student model without original training data.
problem Training without original labeled data.
method Prediction-only distillation scheme for linear and logistic regression.
result Prediction mixing can outperform both the teacher and pure-distilled models.
ESPD improves learning efficiency in sparse reward reinforcement learning.
problem Sparse reward reinforcement learning challenges.
method Evolutionary Stochastic Policy Distillation (ESPD) based on drifted random walk insight.
result High learning efficiency demonstrated in MuJoCo robotics control suite experiments.
The introduction of data analytics into medicine has changed the nature of patient treatment. In this, patients are asked to disclose personal information such as genetic markers, lifestyle habits, and clinical history. This data is then used by statistical models to predict personalized treatments. However, due to pri…
SPEQ improves quantized neural networks by stochastic precision sharing and cosine similarity loss.
problem Improving quantized deep neural networks for edge devices.
method SPEQ combines stochastic precision sharing and cosine similarity loss for knowledge distillation.
result SPEQ outperforms existing methods in various tasks.
FactorMiner discovers financial alpha factors with low redundancy.
problem Finding novel financial alpha factors in a vast search space.
method Modular Skill Architecture and Experience Memory to distill and guide exploration.
result FactorMiner constructs a diverse library of high-quality factors with competitive performance.
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
problem Training robust deep neural networks requires addressing issues like label noise and low entropy predictions.
method ProSelfLC progressively increases trust in predicted labels over time, considering entropy and learning time.
result ProSelfLC demonstrates improved robustness in both clean and noisy settings through empirical validation.
Optimal SD improves ridge regression performance strictly and precisely.
problem Improving ridge regression performance through self-distillation.
method Analyzes unconstrained SD for ridge regression, deriving optimal mixing weight and asymptotic risk.
result Optimal SD strictly improves ridge regression performance, with exact risk equivalents derived.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
ACE improves GFlowNet exploration efficiency by balancing complementary search strategies.
problem Efficient exploration of diverse high-probability regions in GFlowNets.
method Adaptive Complementary Exploration (ACE) trains a separate GFlowNet to search underexplored regions.
result Significantly improves approximation accuracy and diverse state discovery.
Self-PU combines self-training with PU learning for improved binary classification.
problem Binary classification from unlabeled data and few labeled positives.
method Integrates self-paced training, self-calibrated loss, and self-distillation.
result Significantly improved performance on Alzheimer's Disease classification.
The soaring demand for intelligent mobile applications calls for deploying powerful deep neural networks (DNNs) on mobile devices. However, the outstanding performance of DNNs notoriously relies on increasingly complex models, which in turn is associated with an increase in computational expense far surpassing mobile d…
Improved ImageNet classification with semi-supervised learning.
problem Image classification with limited labeled data.
method Noisy Student Training: semi-supervised learning with noisy student models.
result 88.4% top-1 accuracy on ImageNet, 2.0% better than state-of-the-art.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Improved dataset distillation for images and texts boosts model accuracy.
problem Reducing dataset size for faster and more energy-efficient model training.
method Simultaneous distillation of images and soft labels, extending to text datasets.
result 2-4% increase in accuracy for image classification tasks, 20% reduction in distilled samples.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.