Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
Improved dataset distillation for images and texts boosts model accuracy.
problem Reducing dataset size for faster and more energy-efficient model training.
method Simultaneous distillation of images and soft labels, extending to text datasets.
result 2-4% increase in accuracy for image classification tasks, 20% reduction in distilled samples.
Paper tackles label noise in datasets, proposing a robust learning algorithm.
problem Real-world datasets often have label noise.
method Introduces distilled examples and proposes a learning algorithm with theoretical guarantees.
result Classifiers on distilled examples converge to the Bayes optimal classifier under certain conditions.
Efficiently distills white-box adversarial attacks into black-box models.
problem Generating efficient adversarial examples for robustness.
method Train a model to emulate white-box attack behavior and distill it into a more efficient black-box model.
result Reduces adversarial example generation time by 19x-39x and transfers to black-box settings.
This work explores different distillation techniques in reinforcement learning.
problem Understanding and optimizing the transfer of knowledge between policies.
method Theoretical and empirical analysis of various distillation techniques.
result Three preferred distillation techniques are identified, including expected entropy regularised distillation.
Paper improves natural language understanding with less data using a new training method.
problem Limited data hinders performance of small models in natural language tasks.
method Generation-Distillation: uses large finetuned models to generate new training data and distill knowledge into smaller models.
result Achieves comparable performance to BERT with 300x fewer parameters and outperforms prior distillation methods.
Anti-Distillation improves reproducibility of deep networks by making ensemble predictions more diverse.
problem Deep networks are prone to high prediction differences, making them less reproducible.
method Anti-Distillation uses ensembles to force predictions to be more different and diverse.
result Anti-Distillation reduces prediction differences by making ensemble predictions more diverse.
This work improves knowledge distillation by transferring full kernel matrices efficiently.
problem Efficiently transferring full pairwise similarity matrices for model compression in deep learning.
method The authors propose a method to transfer the full similarity matrix effectively using the Nyström method, decomposing it into partial matrices.
result The difference between the full kernel matrices of teacher and student can be well bounded by partial matrices, improving optimization efficiency.
Paper introduces a simulator-free approach to reinforcement learning policy distillation.
problem Learning multiplicity of cases corresponding to a given action in reinforcement learning.
method Generative adversarial approach to find multiple exemplars for each output class.
result Improves over state-of-the-art on data-free learning of student networks.
Paper distills large datasets into smaller, synthetic ones for model training.
problem Training models on large datasets is computationally expensive.
method Dataset distillation: synthesize small, synthetic datasets from large ones.
result Synthetic datasets can approximate performance of models trained on original large datasets.
A method for training neural networks with few data by mimicking predictions.
problem Training neural networks with limited labeled data.
method Pseudo example optimization to mimic predictions of robust reference models.
result The method outperforms other baselines on benchmark datasets.
New method improves knowledge transfer from large to small networks.
problem Improving knowledge transfer from large to small neural networks.
method Contrastive learning to capture more structural knowledge.
result Contrastive learning outperforms knowledge distillation on various transfer tasks.
Machine learning is vulnerable to adversarial examples: inputs carefully modified to force misclassification. Designing defenses against such inputs remains largely an open problem. In this work, we revisit defensive distillation---which is one of the mechanisms proposed to mitigate adversarial examples---to address it…
This paper explores explaining GBDT2NN predictions by its teacher model, improving distillation performance.
problem Explaining GBDT2NN predictions when the models have different structures.
method Empirical study on new approach to explain GBDT2NN predictions and use it as an auxiliary learning task.
result Proposed methods achieve better performance on both explanations and predictions.
Current approaches for Knowledge Distillation (KD) either directly use training data or sample from the training data distribution. In this paper, we demonstrate effectiveness of 'mismatched' unlabeled stimulus to perform KD for image classification networks. For illustration, we consider scenarios where this is a comp…
PS-KD distills a model's own knowledge to soften hard targets during training.
problem Improving generalization of deep neural networks by softening hard targets.
method Progressive self-knowledge distillation (PS-KD) that progressively distills a model's own knowledge to soften hard targets.
result PS-KD improves accuracy and provides high quality of confidence estimates in terms of calibration and ordinal ranking.
Paper introduces CoD for few-shot task-aware knowledge distillation using counterfactual explanations.
problem Lack of data for task-aware distillation in resource-constrained scenarios.
method Counterfactual-explanation-infused Distillation CoD for few-shot task-aware knowledge distillation.
result CoD achieves superior performance with significantly fewer samples than baseline methods.
Fastens diffusion model sampling by distillation.
problem Slow sampling time of diffusion models.
method New parameterizations and progressive distillation.
result Models can be distilled to take half as many sampling steps.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
Extracurricular learning closes the accuracy gap in knowledge distillation.
problem Accuracy gap between teacher and student models after knowledge distillation.
method Modeling student and teacher output distributions, sampling from an extended data distribution, and matching over this set.
result Extracurricular learning reduces the accuracy gap by 46% to 68%.
Bad models can teach well by replicating noise.
problem Overparameterized models can replicate noise in training data.
method Knowledge distillation from noisy samplers.
result Distillation from samplers approximates Bayes optimal classifier.
Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-se…
Proposes learning ordered Top-k attacks for image classification.
problem Vulnerability of DNNs to adversarial attacks, especially white-box targeted attacks.
method Adversarial distillation framework to compute adversarial probability distributions and minimize KL divergence.
result Outperforms C&W method in Top-1 and Top-5 settings for image classification.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Kempe discusses NTK approach to machine learning problems.
problem Generally unsolvable machine learning problems.
method NTK approach focusing on kernel formulations.
result Practical applications like data distillation and adversarial robustness.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Real-time policy distillation speeds up and improves reinforcement learning.
problem Slow and inefficient policy distillation in reinforcement learning.
method Simultaneous training and distillation of a teacher and student model.
result Significantly reduced distillation time and improved small model performance.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
MixKD improves large-scale language model compression and generalization.
problem Inefficient and resource-intensive large-scale language models.
method MixKD uses mixup data augmentation to enhance student model's generalization ability.
result MixKD leads to significant performance gains over standard KD and competitive baselines.
This paper improves robustness in small neural networks through distillation.
problem Vulnerability of small neural networks to adversarial attacks.
method Adversarially Robust Distillation (ARD) to transfer robustness from teacher to student networks.
result ARD produces small models with superior robust accuracy compared to adversarially trained networks.
Deep learning improves brain tumor detection with few data.
problem Scarce training data and data corruption in medical diagnosis.
method Data distillation and augmentation to improve deep neural network training.
result Reaches human expert-level accuracy with just a few thousand training examples.
Distillation works even with hard labels from overparameterized teacher, leading to better performance.
problem Improving model performance with hard labels from overparameterized teacher.
method Training a student model on a large held-out dataset labeled by a highly overparameterized teacher.
result Student model outperforms traditional approaches due to double descent phenomenon.
Defensive distillation fails against targeted adversarial attacks, forcing a tradeoff between learning and security.
problem Defensive distillation's limitations in blocking targeted adversarial attacks.
method Systematic exploration of defensive distillation's effectiveness and limitations.
result Defensive distillation is effective against non-targeted attacks but fails against targeted attacks, necessitating a tradeoff between learning and security.
Expands pre-trained deep networks to classify new classes with minimal data.
problem Learning new classes with limited data.
method Hard distillation with a compact generative model.
result Low-shot network expansion is possible with minimal memory and data.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
OKDDip uses diverse peers to improve online knowledge distillation.
problem Early saturation in group-based distillation.
method Two-level distillation with multiple auxiliary peers and a group leader, using attention-based aggregation weights.
result OKDDip consistently gives better performance than state-of-the-art approaches.
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. Role-wise data augmentation improves knowledge distillation effectiveness.
problem Existing knowledge distillation methods fail to utilize the full potential of teacher-student data interaction.
method Design and implement data augmentation agents with distinct roles for teacher and student.
result Specially tailored data points enhance the demonstration of teacher's knowledge to the student.
Compress U-net by over 1000x with knowledge distillation.
problem Compressing U-net architecture while maintaining performance.
method Knowledge distillation with regularization methods.
result Compressed U-net by 1000x with negligible performance drop.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
RONA compresses complex models while ensuring privacy.
problem Deploying complex deep neural networks on mobile devices poses privacy risks and computational constraints.
method RONA uses knowledge distillation, hint learning, and self learning to train a compact neural network with differential privacy guarantees.
result RONA achieves 20x compression and 19x speed-up with 0.97% accuracy loss on SVHN while maintaining strong privacy.
TEAM generates more powerful adversarial examples for DNNs.
problem Vulnerability of DNNs to imperceptible adversarial examples.
method TEAM uses Taylor expansion and Lagrangian multiplier method to craft adversarial examples.
result TEAM generates adversarial examples with 100% attack success rate using smaller perturbations.
Paper proposes a method to protect DCNNs from adversarial attacks and poor quality images.
problem DCNNs are vulnerable to adversarial examples and perceptual quality issues.
method Combines image quality assessment with knowledge distillation.
result Detector effectively identifies poor quality and adversarial images.
Distiller simplifies DNN compression research with a Python package.
problem Efficiently compressing deep neural networks.
method Open-source Python package with DNN compression algorithms.
result Facilitates new research and learning tasks in DNN compression.
The paper stabilizes tree-based explanations for machine learning models.
problem Stability of tree-based explanations in machine learning models.
method Develops tests and stopping rules to ensure stability of decision tree explanations.
result Ensures that tree-based explanations are robust to small changes in training data.