AC-Teach uses an ensemble of suboptimal teachers to improve exploration in RL.
problem Improving exploration efficiency in long-horizon tasks with sparse rewards.
method Bayesian Actor-Critic with an ensemble of suboptimal teachers.
result AC-Teach improves sample efficiency over baselines on various tasks.
Paper distills word embeddings to reduce dimensionality without sacrificing accuracy.
problem Reducing neural model size for practical deployment.
method Teacher-student model-based embedding distillation with ensemble learning.
result Significant reduction in model size (80x faster and lighter) with minimal accuracy loss.
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change …
Novel framework learns efficient student models from teacher networks.
problem Model capacity gap between teacher and student networks.
method Neural architecture search and oracle knowledge distillation.
result Searched student models often outperform teacher models.
The rapid adoption of machine learning has increased concerns about the privacy implications of machine learning models trained on sensitive data, such as medical records or other personal information. To address those concerns, one promising approach is Private Aggregation of Teacher Ensembles, or PATE, which transfer…
SPEQ improves quantized neural networks by stochastic precision sharing and cosine similarity loss.
problem Improving quantized deep neural networks for edge devices.
method SPEQ combines stochastic precision sharing and cosine similarity loss for knowledge distillation.
result SPEQ outperforms existing methods in various tasks.
Distilled models often fail to match teacher models, despite improving generalization.
problem The discrepancy between teacher and student predictive distributions remains large.
method Investigated the optimization difficulties and dataset details affecting student performance.
result Optimizing for matching the teacher does not always lead to better generalization.
New method improves knowledge transfer from large to small networks.
problem Improving knowledge transfer from large to small neural networks.
method Contrastive learning to capture more structural knowledge.
result Contrastive learning outperforms knowledge distillation on various transfer tasks.
Bayesian Dark Knowledge fails to perform well with high posterior uncertainty.
problem Performance degradation of Bayesian Dark Knowledge with high posterior uncertainty.
method Compresses posterior predictive distribution into a single network, using a student network matching the teacher ensemble architecture.
result Using a matching student network architecture does not guarantee acceptable performance with high posterior uncertainty.
Graph convolutional network (GCN) provides a powerful means for graph-based semi-supervised tasks. However, as a localized first-order approximation of spectral graph convolution, the classic GCN can not take full advantage of unlabeled data, especially when the unlabeled node is far from labeled ones. To capitalize on…
G-PATE generates private data with high utility using teacher-discriminator aggregation.
problem Privacy concerns in large-scale data sharing for machine learning.
method Generative adversarial nets combined with private gradient aggregation among discriminators.
result Significantly improves privacy budget efficiency and data utility.
The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data…
This paper improves speech recognition by distilling knowledge from acoustic models.
problem Improving speech recognition accuracy using ensemble models.
method Proposes multi-teacher distillation strategies for joint CTC-attention end-to-end ASR systems, integrating error rate metric for optimization.
result Reports state-of-the-art error rates on various datasets and languages.
Some machine learning applications involve training data that is sensitive, such as the medical histories of patients in a clinical trial. A model may inadvertently and implicitly store some of its training data; careful analysis of the model may therefore reveal sensitive information. To address this problem, we demon…
This paper adapts PATE for semantic segmentation while maintaining privacy.
problem Preserving privacy in medical machine learning, especially for sensitive information.
method Adapting PATE for semantic segmentation using low-dimensional representations and low-sensitivity queries.
result An Autoencoder-based PATE variant achieves a higher Dice coefficient for the same privacy guarantee.
Study explores various KD methods to improve model performance.
problem Improving model performance with limited data.
method Nine different KD methods covering various aspects of the teacher.
result KD framework provides generalization gains over standard training.
Active learning aims to reduce annotation cost by predicting which samples are useful for a human teacher to label. However it has become clear there is no best active learning algorithm. Inspired by various philosophies about what constitutes a good criteria, different algorithms perform well on different datasets. Th…
Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS) which is able to find the best convolutional layer or cell that is then used as a building block for the network. However, once a good buil…
Proposes RaT to mitigate bias in student-teacher estimation.
problem Systematic bias in teacher's predictions propagates to student model.
method Uses teacher to estimate residuals in student's predictions.
result RaT method reduces teacher bias effect and achieves optimal rate.
OKDDip uses diverse peers to improve online knowledge distillation.
problem Early saturation in group-based distillation.
method Two-level distillation with multiple auxiliary peers and a group leader, using attention-based aggregation weights.
result OKDDip consistently gives better performance than state-of-the-art approaches.
Enhances model compression with multi-teacher knowledge distillation.
problem Uncertainty evaluation and diverse teacher expertise in model deployment.
method Bayesian inference and teacher-informed prior with entropy-based weighting.
result Improved predictive accuracy and robust uncertainty quantification.
The paper offers a simple proof of Condorcet's jury theorem.
problem The relationship between majority voting and Condorcet's jury theorem.
method A simple derivation of Condorcet's jury theorem.
result Condorcet's jury theorem is more likely to choose correctly when individual votes are often correct and independent.
Aims to learn from multiple unpredictable teachers with minimal interaction.
problem Learning from multiple non-deterministic teachers with low interaction cost.
method Develops a framework and an active learning algorithm to estimate a distribution over policy space.
result Significantly reduces interaction with teachers without compromising performance.
Student-teacher learning improves generalization with noisy inputs.
problem Transfer knowledge from clean inputs to noisy inputs.
method Analyzes student-teacher learning using deep linear networks and experiments with nonlinear networks.
result Three factors are vital for success: zero training loss, teacher knowledge, and feature decomposition.
Paper introduces efficient uncertainty estimation in LLMs without multiple forward passes.
problem Accurate uncertainty quantification in LLMs remains challenging.
method Evidential Knowledge Distillation to create compact student models.
result Efficient uncertainty estimation achieved with single forward pass.
Deep neural networks learn spatially heterogeneous patterns from input data.
problem Understanding the hidden layers of deep neural networks.
method Statistical mechanics approach with a teacher-student setting.
result Learning by deep neural networks is spatially heterogeneous, with central regions less correlated.
In this paper, we make an important step towards the black-box machine teaching by considering the cross-space machine teaching, where the teacher and the learner use different feature representations and the teacher can not fully observe the learner's model. In such scenario, we study how the teacher is still able to …
Subclass distillation improves small models by matching teacher's subclass probabilities.
problem Improving small models trained on limited data.
method Train a small model to match probabilities of subclasses invented by a large teacher model.
result Better small models trained on limited data.
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
problem Overwhelming size of Deep Neural Networks (DNNs).
method Single teacher with multiple student networks, transferring knowledge to each student.
result The combined knowledge of the class of students achieves better performance and reduces parameters.
The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student …
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Teaches manifolds from teacher's structured data.
problem Learning a manifold from teacher's structured data.
method Extends existing approaches to learning from randomly sampled data points, considering structured data provided by a teacher.
result Demonstrations can significantly reduce data points needed for manifold learning.
A new method transfers knowledge without data, matching teacher's predictions closely.
problem Lack of access to training data for knowledge transfer.
method Adversarial training to match teacher's predictions without data.
result Zero-shot student performs well on CIFAR10, improving state-of-the-art.
Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teac…
Random feature models can outperform a weak teacher with early stopping.
problem Generalization from a weak to a strong model in random feature networks.
method Random feature models, early stopping, proving weak-to-strong generalization.
result Random feature models can outperform a weak teacher with early stopping.
This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.
problem Teacher hacking during language model distillation, leading to suboptimal performance.
method A controlled experimental setup involving an oracle LM, teacher LM, and student LM, using fixed offline or online data generation techniques.
result Data diversity is the key factor in preventing teacher hacking during distillation.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Machine teaching addresses the problem of finding the best training data that can guide a learning algorithm to a target model with minimal effort. In conventional settings, a teacher provides data that are consistent with the true data distribution. However, for sequential learners which actively choose their queries,…
KD can lead to student-teacher deviations that improve performance.
problem KD can lead to student-teacher deviations that may outperform the teacher.
method Characterized and explained the nature of student-teacher deviations through experiments and theory.
result KD can lead to improved generalization by exaggerating the implicit bias of gradient descent.
Role-wise data augmentation improves knowledge distillation effectiveness.
problem Existing knowledge distillation methods fail to utilize the full potential of teacher-student data interaction.
method Design and implement data augmentation agents with distinct roles for teacher and student.
result Specially tailored data points enhance the demonstration of teacher's knowledge to the student.
A new KD layer lets student models learn and apply teacher knowledge explicitly.
problem Implicit action of traditional KD on student's feature transform limits its use in intermediate layers.
method Proposes a learnable KD layer that explicitly embeds teacher's knowledge in feature transform.
result Improves KD with two abilities: leveraging teacher's knowledge and feeding forward knowledge deeper.
Improved learning to reweight using deep interactions between student and teacher models.
problem Limitation of existing learning to reweight methods in utilizing student model's internal states.
method Proposes an algorithm that uses the student model's internal states to the teacher model, which returns adaptive weights to enhance student model training.
result Significant improvement over previous methods in image classification and neural machine translation experiments.
The paper reveals three mechanisms for weak-to-strong generalization.
problem Understanding the mechanisms behind weak-to-strong generalization in imperfect labeling scenarios.
method Theoretical analysis of simple models including ridge regression and weighted ridge regression, and a nonlinear multi-index setting.
result A student model can compensate for a teacher's under-regularization and achieve lower test error.
This work proposes a student-teacher network for predicting hospital admission locations.
problem Accurate prediction of hospital admission locations to optimize resource allocation.
method Reinforcement learning approach where a teacher network selects data batches for a student network.
result The approach outperforms state-of-the-art methods on tabular data and image recognition.