arXiv research

Proposes LsrKD and MrKD to improve neural network training performance.

Recent variants improve knowledge distillation performance.

Survey of knowledge distillation for resource-limited devices.

This study improves knowledge distillation for RNN-T models with noisy labels.

RID framework quantifies and regularizes task-relevant knowledge in distillation.

The paper explains knowledge distillation by analyzing visual concepts in DNNs.

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a g…

CAKD framework optimizes knowledge transfer by focusing on influential components of distillation.

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approa…

A new distillation method transfers channel information from teacher to student.

Knowledge distillation improves model accuracy by mimicking teacher model probabilities.

Study on ensemble, distillation, and self-distillation in deep learning models.

Survey of knowledge distillation for model compression.

A new method for training GNNs without a teacher model.

PLD distills knowledge using choice-theoretic Plackett-Luce model.

We present a novel framework of knowledge distillation that is capable of learning powerful and efficient student models from ensemble teacher networks. Our approach addresses the inherent model capacity issue between teacher and student and aims to maximize benefit from teacher models during distillation by reducing t…

Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget. It is a commonly used technique for model compression, where a larger capacity teacher model with better quality is used to train a more compact student model with better inference efficiency. Throug…

We study the use of knowledge distillation to compress the U-net architecture. We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves …

Two approaches extend knowledge distillation to Gaussian Processes, showing relationships to existing methods.

Knowledge distillation boosts simple models in banking without complexity.

New method regularizes deep networks by distilling self-knowledge.

This paper improves speech recognition by distilling knowledge from acoustic models.

The paper analyzes knowledge distillation in wide neural networks, providing theoretical insights and practical implications.

Enhances model compression with multi-teacher knowledge distillation.

Distilled models often fail to match teacher models, despite improving generalization.

Paper introduces CoD for few-shot task-aware knowledge distillation using counterfactual explanations.

A modular framework for knowledge distillation simplifies experiments and reproducibility.

Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student. However, just like massive classification, large scale knowledge distillation also imposes heavy computational costs on training models of deep neural networks, as the softmax activations at the la…

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which…

Distillation affects some classes more than others, impacting fairness and bias.

KD$^{2}$M unifies feature matching in neural networks.

Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate pr…

Distillation speeds up classifier training and provides insights into its success.

Distillation improves simple models by approximating complex labels.

Extracurricular learning closes the accuracy gap in knowledge distillation.

This paper improves ASR performance by aligning frames more accurately.

Paper explores how knowledge distillation transfers inductive biases between models.

FairDTD improves fairness in GNNs by distilling dual teacher knowledge, balancing utility and bias.

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learnin…

Knowledge distillation (KD) is one of the most potent ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To re…

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robu…

Paper introduces a simulator-free approach to reinforcement learning policy distillation.

New method distills cloud models into edge-friendly ones.

The paper establishes theoretical foundations for low-rank knowledge distillation in LLMs.

Hybrid approach combines Markowitz's theory with reinforcement learning for optimal portfolio management.

A new algorithm improves federated learning by combining knowledge distillation and weighted combination loss.

This work proposes splitting deep neural networks into smaller sub-networks for faster and more efficient distillation.

Role-wise data augmentation improves knowledge distillation effectiveness.