Student-teacher learning improves generalization with noisy inputs.
problem Transfer knowledge from clean inputs to noisy inputs.
method Analyzes student-teacher learning using deep linear networks and experiments with nonlinear networks.
result Three factors are vital for success: zero training loss, teacher knowledge, and feature decomposition.
Proposes RaT to mitigate bias in student-teacher estimation.
problem Systematic bias in teacher's predictions propagates to student model.
method Uses teacher to estimate residuals in student's predictions.
result RaT method reduces teacher bias effect and achieves optimal rate.
Self-training with noisy student-teacher boosts keyword spotting accuracy.
problem Robust keyword spotting in challenging conditions.
method Aggressive data augmentation and self-training with noisy student-teacher approach.
result Significant accuracy improvement in difficult conditions, up to 60%.
A new method transfers knowledge without data, matching teacher's predictions closely.
problem Lack of access to training data for knowledge transfer.
method Adversarial training to match teacher's predictions without data.
result Zero-shot student performs well on CIFAR10, improving state-of-the-art.
Improved learning to reweight using deep interactions between student and teacher models.
problem Limitation of existing learning to reweight methods in utilizing student model's internal states.
method Proposes an algorithm that uses the student model's internal states to the teacher model, which returns adaptive weights to enhance student model training.
result Significant improvement over previous methods in image classification and neural machine translation experiments.
A new training method improves MLIPs for faster, lighter simulations.
problem High computational and memory costs of complex MLIPs for large-scale MD simulations.
method Teacher-student training framework using latent atomic energy knowledge.
result Lightweight student MLIPs achieve faster MD speeds and comparable accuracy to teachers.
Subclass distillation improves small models by matching teacher's subclass probabilities.
problem Improving small models trained on limited data.
method Train a small model to match probabilities of subclasses invented by a large teacher model.
result Better small models trained on limited data.
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
problem Overwhelming size of Deep Neural Networks (DNNs).
method Single teacher with multiple student networks, transferring knowledge to each student.
result The combined knowledge of the class of students achieves better performance and reduces parameters.
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Improved knowledge distillation using a teacher assistant to bridge the gap between student and teacher networks.
problem Large neural networks are hard to deploy on edge devices due to size constraints.
method Introduce multi-step knowledge distillation with an intermediate-sized teacher assistant.
result The proposed multi-step distillation method improves student network performance.
This paper proposes a method to embed teacher knowledge into a student network without increasing parameters.
problem The need for portable neural networks on mobile devices with limited resources.
method Feature embedding approach to distill knowledge from a teacher network to a student network without introducing new parameters.
result The proposed method maintains the performance of the teacher network while significantly reducing computational and storage complexity.
KD can lead to student-teacher deviations that improve performance.
problem KD can lead to student-teacher deviations that may outperform the teacher.
method Characterized and explained the nature of student-teacher deviations through experiments and theory.
result KD can lead to improved generalization by exaggerating the implicit bias of gradient descent.
This work proposes a student-teacher network for predicting hospital admission locations.
problem Accurate prediction of hospital admission locations to optimize resource allocation.
method Reinforcement learning approach where a teacher network selects data batches for a student network.
result The approach outperforms state-of-the-art methods on tabular data and image recognition.
Student network specializes teacher nodes in deep ReLU networks.
problem Training deep ReLU networks with finite width and input dimension.
method Stochastic Gradient Descent (SGD) on over-realized student network trained from teacher network output.
result Each teacher node is specialized by at least one student node at the lowest layer under mild conditions.
Distilled models often fail to match teacher models, despite improving generalization.
problem The discrepancy between teacher and student predictive distributions remains large.
method Investigated the optimization difficulties and dataset details affecting student performance.
result Optimizing for matching the teacher does not always lead to better generalization.
Knowledge flow transfers knowledge from multiple teachers to a student net.
problem Choosing and initializing deep nets for new tasks is unclear and inefficient.
method Develops a method to move 'knowledge' from multiple deep nets (teachers) to a new net (student) without dependency on teachers.
result The student net outperforms fine-tuning and other methods on various tasks.
Role-wise data augmentation improves knowledge distillation effectiveness.
problem Existing knowledge distillation methods fail to utilize the full potential of teacher-student data interaction.
method Design and implement data augmentation agents with distinct roles for teacher and student.
result Specially tailored data points enhance the demonstration of teacher's knowledge to the student.
Proposes KTAN for better training of student networks with both intermediate representations and probability distributions.
problem Reduces large computation and storage cost of deep networks by transferring generalization ability.
method Holistically considers intermediate representations and probability distributions; uses a Teacher-to-Student layer and adversarial learning.
result Significantly improves performance of student networks on image classification and object detection tasks.
Improved machine learning models outperform their simpler counterparts by using imperfect labels.
problem Improving model performance using imperfect labels.
method Random feature ridge regression (RFRR) with a deterministic equivalent for excess test error.
result The student model can outperform the teacher model regardless of the teacher's scaling law, achieving the minimax optimal rate.
A student classifier learns from feature visualizations of a teacher network to achieve high accuracy.
problem Machine learning-based sleep apnea detection with limited access to training data.
method Interpretation-based indirect knowledge transfer using activation maximization and synthetic datasets.
result The student classifier achieves 97.8% accuracy on MNIST and 86.1-89.5% on Apnea-ECG dataset.
Random feature models can outperform a weak teacher with early stopping.
problem Generalization from a weak to a strong model in random feature networks.
method Random feature models, early stopping, proving weak-to-strong generalization.
result Random feature models can outperform a weak teacher with early stopping.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
This paper improves student networks to be more robust against perturbations.
problem Improving robustness of lightweight student networks.
method The paper introduces a method to make student networks more confident and robust by leveraging teacher networks.
result The proposed method enhances the robustness of student networks without sacrificing accuracy.
A new method for distilling predictions from a teacher model to a student model without original training data.
problem Training without original labeled data.
method Prediction-only distillation scheme for linear and logistic regression.
result Prediction mixing can outperform both the teacher and pure-distilled models.
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
This work improves ASR noise robustness using parallel data and T/S learning.
problem Noise robustness in automatic speech recognition.
method Teacher-student learning with parallel clean and noisy data, logits selection.
result Best student model yields significant WER reductions in noisy conditions.
The paper reveals three mechanisms for weak-to-strong generalization.
problem Understanding the mechanisms behind weak-to-strong generalization in imperfect labeling scenarios.
method Theoretical analysis of simple models including ridge regression and weighted ridge regression, and a nonlinear multi-index setting.
result A student model can compensate for a teacher's under-regularization and achieve lower test error.
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4imes faster per training step. Two-layer ReLU networks outperform kernel methods in teacher-student settings.
problem Understanding the excess risk of two-layer ReLU neural networks in teacher-student models.
method Investigated a two-phase training process for a student network, comparing it to kernel methods.
result The student network reaches near-global optimality and outperforms kernel methods in minimax optimal rate.
LIT compresses deep networks by training intermediate representations, outperforming traditional methods.
problem Reducing the computational overhead of deep network inference.
method LIT trains a student model with the same width but shallower depth, using the intermediate representations from the teacher model.
result LIT achieves substantial network depth reductions without accuracy loss, outperforming traditional methods.
Knowledge transfer speeds up neural classifier training.
problem Lack of theoretical analysis of knowledge transfer in neural networks.
method Regularization of fit between teacher and student networks using privileged information.
result Wide two-layer networks can interpolate between privileged information and data, improving generalization.
A new KD layer lets student models learn and apply teacher knowledge explicitly.
problem Implicit action of traditional KD on student's feature transform limits its use in intermediate layers.
method Proposes a learnable KD layer that explicitly embeds teacher's knowledge in feature transform.
result Improves KD with two abilities: leveraging teacher's knowledge and feeding forward knowledge deeper.
Paper generalizes teacher-student model for realistic data.
problem Capturing learning curves for realistic datasets.
method Introduces a Gaussian covariate generalization of the teacher-student model.
result Generalized model captures learning curves for various realistic data sets.
Some machine learning applications involve training data that is sensitive, such as the medical histories of patients in a clinical trial. A model may inadvertently and implicitly store some of its training data; careful analysis of the model may therefore reveal sensitive information. To address this problem, we demon…
Distillation works even with hard labels from overparameterized teacher, leading to better performance.
problem Improving model performance with hard labels from overparameterized teacher.
method Training a student model on a large held-out dataset labeled by a highly overparameterized teacher.
result Student model outperforms traditional approaches due to double descent phenomenon.
GAN-TSC improves student model accuracy by augmenting with synthetic data.
problem Suboptimal compression when fresh data is unavailable for TSC.
method Augment compression dataset with synthetic data from a GAN.
result Significant improvement in student accuracy for expensive models.
Study on SGD dynamics in neural networks, revealing generalisation patterns.
problem Understanding generalisation in over-parameterised neural networks.
method Analysis of SGD dynamics in a teacher-student setup using differential equations.
result Network size affects generalisation error differently depending on training layers and activation functions.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
A new method trains compact ranking models for recommender systems.
problem Training efficient ranking models for recommender systems.
method Ranking Distillation (RD) technique to train a smaller model with similar performance to a larger teacher model.
result The student model achieves a similar ranking performance to the teacher model with significantly less model size.
Knowledge distillation improves model accuracy by mimicking teacher model probabilities.
problem Improving model accuracy through model compression.
method Casting knowledge distillation as a semiparametric inference problem, deriving new guarantees, and developing enhancements.
result Enhancements improve student performance by mitigating teacher overfitting and underfitting.
A new method identifies a stable subnetwork in overparameterized student models.
problem Identifying a stable subnetwork in overparameterized student models.
method Spectral representation of linear transfer of information, focusing on eigenvalues and eigenvectors.
result A stable student substructure is isolated that mirrors the true complexity of the teacher.
Teaches AI models to learn effectively through dynamic loss functions.
problem Optimizing machine learning models' performance through dynamic loss functions.
method Develops a method for a teacher model to dynamically output loss functions for a student model, enabling gradient-based optimization.
result Significantly improves the performance of various student models in real-world tasks.
Improved ImageNet classification with semi-supervised learning.
problem Image classification with limited labeled data.
method Noisy Student Training: semi-supervised learning with noisy student models.
result 88.4% top-1 accuracy on ImageNet, 2.0% better than state-of-the-art.
A new method trains a smaller model from a larger one without needing the actual training data.
problem Training a smaller model from a larger one without access to the training data.
method Synthesizes data impressions from the Teacher model to train the Student model.
result Zero-Shot Knowledge Distillation achieves competitive generalization performance.
A new method transfers adversarial robustness from teacher to student using feature distillation.
problem Adversarial robustness transfer across different models and tasks.
method Guided Adversarial Contrastive Distillation (GACD) with contrastive learning and sample reweighted estimation.
result GACD effectively transfers adversarial robustness from teacher to student, achieving comparable or better results.
New approach uses synthetic labels to train models on scarce annotated data for surgical phase recognition.
problem Learning surgical phase recognition from limited annotated data.
method Teacher/Student approach with a CNN-biLSTM-CRF teacher generating synthetic labels for a CNN-LSTM student.
result Improved surgical phase recognition performance with fewer annotated videos.
This paper improves understanding of knowledge distillation and its effects.
problem Improving model quality with a fixed capacity budget.
method Categorizing teacher's knowledge into three levels and studying their effects on distillation.
result The three hierarchical levels of teacher's knowledge (universe, domain, instance) significantly impact knowledge distillation.