A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Aims to learn from multiple unpredictable teachers with minimal interaction.
Student-teacher learning improves generalization with noisy inputs.
Enhances model compression with multi-teacher knowledge distillation.
The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student …
Study shows how task similarity affects forgetting in teacher-student setup.
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
This work proposes a student-teacher network for predicting hospital admission locations.
Improved learning to reweight using deep interactions between student and teacher models.
Paper proposes IRL methods for limited interaction scenarios.
The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency. Thus, improving over random exploration is crucial to solve long-horizon tasks with sparse rewards. We propose to leverage an ensemble of partial solutions as teachers that guide the agent's…
We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher. More formally, we tackle the following algorithmic question: How could a teacher provide an informative sequence of demonstrations to an IRL learner to speed up the learning process? We p…
A new KD layer lets student models learn and apply teacher knowledge explicitly.
After a large "teacher" neural network has been trained on labeled data, the probabilities that the teacher assigns to incorrect classes reveal a lot of information about the way in which the teacher generalizes. By training a small "student" model to match these probabilities, it is possible to transfer most of the ge…
A new method transfers adversarial robustness from teacher to student using feature distillation.
A new distillation method transfers channel information from teacher to student.
We consider the problem of learning a manifold from a teacher's demonstration. Extending existing approaches of learning from randomly sampled data points, we consider contexts where data may be chosen by a teacher. We analyze learning from teachers who can provide structured data such as individual examples (isolated …
In this paper, we make an important step towards the black-box machine teaching by considering the cross-space machine teaching, where the teacher and the learner use different feature representations and the teacher can not fully observe the learner's model. In such scenario, we study how the teacher is still able to …
Machine teaching addresses the problem of finding the best training data that can guide a learning algorithm to a target model with minimal effort. In conventional settings, a teacher provides data that are consistent with the true data distribution. However, for sequential learners which actively choose their queries,…
Estimates model performance from compute budget for distillation.
The paper reveals three mechanisms for weak-to-strong generalization.
Paper generalizes teacher-student model for realistic data.
A new training method improves MLIPs for faster, lighter simulations.
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
Paper analyzes how neural networks learn from a teacher in a specific setting.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
Study on teaching with imperfect knowledge, showing its impact on optimal teaching sets.
Improved machine learning models outperform their simpler counterparts by using imperfect labels.
FairDTD improves fairness in GNNs by distilling dual teacher knowledge, balancing utility and bias.
Role-wise data augmentation improves knowledge distillation effectiveness.
Meta Pseudo Labels boosts image classification accuracy to 90.2%.
We present a novel framework of knowledge distillation that is capable of learning powerful and efficient student models from ensemble teacher networks. Our approach addresses the inherent model capacity issue between teacher and student and aims to maximize benefit from teacher models during distillation by reducing t…
Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel …
Random feature models can outperform a weak teacher with early stopping.
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
Proposes RaT to mitigate bias in student-teacher estimation.
We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and th…
This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.
Method teaches students without teachers, estimating true labels from crowdsourcing.
Knowledge Distillation (KD) consists of transferring âknowledgeâ from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the studentâs…
The rapid adoption of machine learning has increased concerns about the privacy implications of machine learning models trained on sensitive data, such as medical records or other personal information. To address those concerns, one promising approach is Private Aggregation of Teacher Ensembles, or PATE, which transfer…
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change …
Teaching is critical to human society: it is with teaching that prospective students are educated and human civilization can be inherited and advanced. A good teacher not only provides his/her students with qualified teaching materials (e.g., textbooks), but also sets up appropriate learning objectives (e.g., course pr…
We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments. To do so, we study how a teacher algorithm can learn to generate a learning curriculum, whereby it sequentially samples parameters contr…
Optimal transport metric transfers deep network representations efficiently.
Dual active learning improves RLHF by selecting optimal conversations and teachers.
Some machine learning applications involve training data that is sensitive, such as the medical histories of patients in a clinical trial. A model may inadvertently and implicitly store some of its training data; careful analysis of the model may therefore reveal sensitive information. To address this problem, we demon…