PS-KD distills a model's own knowledge to soften hard targets during training.
problem Improving generalization of deep neural networks by softening hard targets.
method Progressive self-knowledge distillation (PS-KD) that progressively distills a model's own knowledge to soften hard targets.
result PS-KD improves accuracy and provides high quality of confidence estimates in terms of calibration and ordinal ranking.
Expands pre-trained deep networks to classify new classes with minimal data.
problem Learning new classes with limited data.
method Hard distillation with a compact generative model.
result Low-shot network expansion is possible with minimal memory and data.
Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
The paper analyzes knowledge distillation in wide neural networks, providing theoretical insights and practical implications.
problem Lack of theoretical understanding of knowledge distillation in wide neural networks.
method Theoretical analysis of knowledge distillation in a linearized model of a wide neural network, introducing a metric of task training difficulty.
result For a perfect teacher, a high ratio of teacher's soft labels can be beneficial. For imperfect teacher, hard labels can correct wrong predictions.
Distillation works even with hard labels from overparameterized teacher, leading to better performance.
problem Improving model performance with hard labels from overparameterized teacher.
method Training a student model on a large held-out dataset labeled by a highly overparameterized teacher.
result Student model outperforms traditional approaches due to double descent phenomenon.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Improved dataset distillation for images and texts boosts model accuracy.
problem Reducing dataset size for faster and more energy-efficient model training.
method Simultaneous distillation of images and soft labels, extending to text datasets.
result 2-4% increase in accuracy for image classification tasks, 20% reduction in distilled samples.
New method distills cloud models into edge-friendly ones.
problem Cloud-to-edge model compression with limited data exchange.
method Two-step workflow of deprivatization and distillation.
result Outperforms previous state-of-the-art approaches on various benchmarks.
TREK uses distillation to help students solve hard problems.
problem Stalled progress on hard prompts when current policy lacks useful reasoning trajectories.
method TREK combines distillation and reinforcement learning to expand student support.
result TREK significantly improves student performance on mathematical reasoning and agentic tasks.
New PL method improves ASR accuracy without pseudo-labels.
problem Improving ASR accuracy with limited labeled data.
method End-to-end continuous pseudo-labeling with soft-labels.
result Soft-labels can lead to model collapse, but regularization can mitigate this.
Knowledge distillation simplifies deep models into interpretable decision trees.
problem Interpretability of deep neural networks is challenging and important for practical deployment.
method Knowledge distillation applied to transform deep models into decision trees.
result The student model achieves better accuracy than vanilla decision trees.
Paper proposes a data-free adversarial distillation method.
problem Tackles the challenge of model compression without real-world data.
method Introduces a novel adversarial distillation mechanism to craft a compact student model without any real-world data.
result Demonstrates comparable performance to data-driven methods and achieves state-of-the-art results in semantic segmentation.
Combines experience replay and exploration for better agent performance.
problem Improving exploration efficiency and robustness in reinforcement learning.
method Integrates Intrinsic Rewards with Prioritized Oversampled Experience Replay (POER).
result Achieves better agent performance and sample efficiency compared to PPO/RND.
Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well…
This work compresses BERT into simple neural networks using unlabeled data.
problem Compression of large BERT models for practical use in downstream tasks.
method Leverage unlabeled transfer data to distill BERT into simple RNN models with hard and soft distillation.
result Simple RNN models can match or exceed BERT performance with up to 26x parameter compression.
Rectified decision trees improve machine learning interpretability and effectiveness.
problem Combining interpretability and effectiveness in machine learning models.
method Knowledge distillation and modified decision tree splitting criteria.
result Soft labels improve model performance and reduce model size.
Study shows optimal self-distillation improves model performance on noisy data.
problem Improving model performance on noisy Gaussian mixture data.
method Hyperparameter-tuned multi-stage self-distillation with a linear classifier, using replica method.
result Primary driver of SD's performance improvement is denoising through hard pseudo-labels.
A new method for exploration in reinforcement learning improves performance on Atari games.
problem Improving exploration in reinforcement learning algorithms, especially for complex games.
method Random Network Distillation (RND) bonus combined with flexible reward combination.
result Significant progress on hard exploration Atari games, including Montezuma's Revenge.
The paper studies how search and distillation improve reasoning in large language models.
problem Improving reasoning capabilities of large language models.
method Viewing chain-of-thought generation as a metastable Markov process, proving benefits of search and distillation.
result Search protocol rewards sparse edges, reducing the expected number of steps to reach different clusters.
Label smoothing improves model calibration and generalization but harms distillation.
problem Understanding the effects of label smoothing on model calibration and distillation.
method Empirical evaluation and visualization of network representations.
result Label smoothing improves model calibration but harms knowledge distillation.
Deep metric learning detects anomalies without labels.
problem Unsupervised anomaly detection for high-dimensional data.
method Deep metric learning with end-to-end optimization, data distillation, hard mining.
result Significant performance gains over state-of-the-art methods.
Graph representation learning improves with domain knowledge.
problem Efficiently learning graph representations from scarce labels.
method Multi-task knowledge distillation combining graph metrics.
result Improves prediction performance, especially with limited training data.
Learning shrinks hard tail, improving inference performance.
problem Improving inference performance in neural networks.
method Latent Instance Difficulty (LID) model analyzing fine-tuning of neural networks.
result Training-dependent inference scaling, with βexteff growing with sample size before saturating. Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of …
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Real-time policy distillation speeds up and improves reinforcement learning.
problem Slow and inefficient policy distillation in reinforcement learning.
method Simultaneous training and distillation of a teacher and student model.
result Significantly reduced distillation time and improved small model performance.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
This paper improves robustness in small neural networks through distillation.
problem Vulnerability of small neural networks to adversarial attacks.
method Adversarially Robust Distillation (ARD) to transfer robustness from teacher to student networks.
result ARD produces small models with superior robust accuracy compared to adversarially trained networks.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. This work explores different distillation techniques in reinforcement learning.
problem Understanding and optimizing the transfer of knowledge between policies.
method Theoretical and empirical analysis of various distillation techniques.
result Three preferred distillation techniques are identified, including expected entropy regularised distillation.
OKDDip uses diverse peers to improve online knowledge distillation.
problem Early saturation in group-based distillation.
method Two-level distillation with multiple auxiliary peers and a group leader, using attention-based aggregation weights.
result OKDDip consistently gives better performance than state-of-the-art approaches.
Compress U-net by over 1000x with knowledge distillation.
problem Compressing U-net architecture while maintaining performance.
method Knowledge distillation with regularization methods.
result Compressed U-net by 1000x with negligible performance drop.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
Among other things, we prove the following two topologcal statements about closed hyperbolic 3-manifolds. First, every rational second homology class of a closed hyperbolic 3-manifold has a positve integral multiple represented by an oriented connected closed π1-injectively immersed quasi-Fuchsian subsurface. Second…
Rotation invariant algorithms fail with hard labels sampled from sparse targets.
problem Rotation invariant algorithms fail to learn from hard labels sampled from sparse targets.
method Proving the excess risk of rotation invariant algorithms and proposing a simple non-rotation invariant algorithm.
result Rotation invariant algorithms incur an excess risk of $Ω\left(\frac{d-1}{n}
ight)$, while non-rotation invariant algorithms have an excess risk of $O\left(\frac{s\log d}{n}
ight).
Distiller simplifies DNN compression research with a Python package.
problem Efficiently compressing deep neural networks.
method Open-source Python package with DNN compression algorithms.
result Facilitates new research and learning tasks in DNN compression.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
FasterVoiceGrad speeds up VC by 6-7x with novel distillation.
problem Slow iterative sampling in diffusion-based VC models.
method Adversarial diffusion conversion distillation (ADCD) to create a faster one-step model.
result 6.6-6.9 and 1.8x faster on GPU and CPU, respectively.
This paper improves speech recognition by distilling knowledge from acoustic models.
problem Improving speech recognition accuracy using ensemble models.
method Proposes multi-teacher distillation strategies for joint CTC-attention end-to-end ASR systems, integrating error rate metric for optimization.
result Reports state-of-the-art error rates on various datasets and languages.
A new KD method distills dataset-based knowledge using MHA.
problem Distilling knowledge from large teacher networks to small student networks.
method Graph-based knowledge distillation by multi-head attention network.
result The method improves SN performance by 7.05% on CIFAR100.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
Novel framework learns efficient student models from teacher networks.
problem Model capacity gap between teacher and student networks.
method Neural architecture search and oracle knowledge distillation.
result Searched student models often outperform teacher models.
CAKD framework optimizes knowledge transfer by focusing on influential components of distillation.
problem Balancing and optimizing knowledge transfer in distillation models.
method Decouple KL divergence into BCD, SCD, and WCD; prioritize influential components.
result CAKD framework consistently outperforms baseline across diverse models and datasets.