GAN-TSC improves student model accuracy by augmenting with synthetic data.
problem Suboptimal compression when fresh data is unavailable for TSC.
method Augment compression dataset with synthetic data from a GAN.
result Significant improvement in student accuracy for expensive models.
Conditional T/S learning improves student model performance by selectively learning from teacher or ground truth.
problem Teacher's occasional wrong guidance leads to suboptimal student model performance.
method Proposes a conditional T/S learning scheme where the student selectively chooses between teacher and ground truth based on teacher correctness.
result The conditional learning achieves significant performance improvements over traditional T/S learning.
Survey of knowledge distillation for model compression.
problem Deploying large deep models on resource-limited devices.
method Knowledge distillation from large teacher models to small student models.
result Effective model compression and acceleration.
RKD improves model compression by distilling residual knowledge from a deep teacher model.
problem Performance degradation due to the gap between student and teacher models.
method Introduces an assistant model to distill residual knowledge from the teacher model.
result RKD achieves better results on popular classification datasets than state-of-the-art methods.
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
A method for semi-supervised sound event detection using teacher-student learning.
problem Weakly-labeled data in sound event detection.
method Guided Learning with a teacher model for audio tagging and a student model for boundary detection.
result The method improves boundary detection performance using unlabeled data.
Proposes a method to compress deep networks using adversarial training and attention mechanisms.
problem Lack of efficient methods to compress deep networks while preserving important information.
method Adversarial training with task-driven attention mechanism for effective intermediate supervision.
result Demonstrates superior performance on benchmark datasets compared to state-of-the-art methods.
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
problem Overwhelming size of Deep Neural Networks (DNNs).
method Single teacher with multiple student networks, transferring knowledge to each student.
result The combined knowledge of the class of students achieves better performance and reduces parameters.
RAF model explains neural networks' dual rule learning and fact memorization.
problem Understanding how neural networks learn rules and memorize facts simultaneously.
method Introduces the Rules-and-Facts (RAF) model to bridge generalization and memorization.
result Characterizes conditions for simultaneous rule learning and fact memorization in neural networks.
Paper generalizes teacher-student model for realistic data.
problem Capturing learning curves for realistic datasets.
method Introduces a Gaussian covariate generalization of the teacher-student model.
result Generalized model captures learning curves for various realistic data sets.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
A novel feature selection method using a teacher-student network.
problem Feature selection for high-dimensional data in machine learning.
method Teacher-student network approach for feature selection.
result The proposed TSFS method outperforms existing methods in classification and clustering.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
problem Understanding the excess risk of two-layer ReLU neural networks in teacher-student models.
method Investigated a two-phase training process for a student network, comparing it to kernel methods.
result The student network reaches near-global optimality and outperforms kernel methods in minimax optimal rate.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
This paper improves understanding of knowledge distillation and its effects.
problem Improving model quality with a fixed capacity budget.
method Categorizing teacher's knowledge into three levels and studying their effects on distillation.
result The three hierarchical levels of teacher's knowledge (universe, domain, instance) significantly impact knowledge distillation.
A new training method improves MLIPs for faster, lighter simulations.
problem High computational and memory costs of complex MLIPs for large-scale MD simulations.
method Teacher-student training framework using latent atomic energy knowledge.
result Lightweight student MLIPs achieve faster MD speeds and comparable accuracy to teachers.
This work improves ASR noise robustness using parallel data and T/S learning.
problem Noise robustness in automatic speech recognition.
method Teacher-student learning with parallel clean and noisy data, logits selection.
result Best student model yields significant WER reductions in noisy conditions.
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
PLD distills knowledge using choice-theoretic Plackett-Luce model.
problem Model compression and knowledge transfer between large and small networks.
method PLD uses a weighted list-wise ranking loss based on the Plackett-Luce model.
result PLD achieves consistent gains across diverse architectures and distillation methods.
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
Randomly sampled interpolators achieve zero generalization error with enough data.
problem Understanding the high generalization ability of machine learning models.
method Algebraic geometry tools to prove zero generalization error for random interpolators.
result Generalization error of randomly sampled interpolators becomes zero once the number of training samples exceeds a geometric threshold.
New approach uses synthetic labels to train models on scarce annotated data for surgical phase recognition.
problem Learning surgical phase recognition from limited annotated data.
method Teacher/Student approach with a CNN-biLSTM-CRF teacher generating synthetic labels for a CNN-LSTM student.
result Improved surgical phase recognition performance with fewer annotated videos.
Deep neural networks outperform traditional methods in high-dimensional classification.
problem Understanding the empirical success of deep neural networks in high-dimensional classification.
method Proposed a teacher-student framework with Bayes classifier as ReLU neural networks, derived convergence rates for 0-1 and hinge losses.
result Sharp rate of convergence for classifiers trained using 0-1 or hinge loss, with Od(n−2/3) or Od(n−1) under separable data distribution. Study online learner attacks by manipulating labels, revealing critical thresholds.
problem Adversarial attacks on online learners, especially label manipulation.
method Control-theoretical perspective, theoretical analysis, empirical study.
result Discontinuous transition in learner's accuracy occurs at a critical threshold.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
New method distills cloud models into edge-friendly ones.
problem Cloud-to-edge model compression with limited data exchange.
method Two-step workflow of deprivatization and distillation.
result Outperforms previous state-of-the-art approaches on various benchmarks.
Study on SGD dynamics in neural networks, revealing generalisation patterns.
problem Understanding generalisation in over-parameterised neural networks.
method Analysis of SGD dynamics in a teacher-student setup using differential equations.
result Network size affects generalisation error differently depending on training layers and activation functions.
This paper proposes a method to embed teacher knowledge into a student network without increasing parameters.
problem The need for portable neural networks on mobile devices with limited resources.
method Feature embedding approach to distill knowledge from a teacher network to a student network without introducing new parameters.
result The proposed method maintains the performance of the teacher network while significantly reducing computational and storage complexity.
Superpixel-mix enhances reliability in semantic segmentation.
problem Improving reliability in real-world semantic segmentation.
method Superpixel-mix, a new data augmentation method with teacher-student consistency training.
result Superpixel-mix achieves state-of-the-art results in semi-supervised semantic segmentation.
Proposes a new theoretical framework for deep locally connected ReLU networks.
problem Understanding theoretical properties of deep locally connected networks.
method Teacher-student setting, explicit formulation of data distribution, disentangled representations, compatibility with regularization techniques.
result Theoretical framework facilitates analysis of practical issues like overfitting and generalization.
Superposition accelerates training to a universal power-law exponent.
problem Training dynamics in neural networks.
method Teacher-student framework and analytic theory.
result Superposition leads to a universal power-law exponent of ~1, independent of data and channel statistics.
Linear generalisation error with network size in neural networks.
problem Understanding generalisation in over-parameterised neural networks.
method Teacher-student setup with SGD dynamics captured by differential equations.
result Generalisation error increases linearly with network size.
Approach for biosensor condition detection using domain adaptation.
problem Limited clinical data for biosensor condition detection.
method Teacher-student adaptation of deep learning models.
result Significantly outperforms other methods on synthetic and real-world datasets.
Linear models can grok without understanding, improving generalization.
problem Understanding the phenomenon of grokking in linear models.
method Analytical and numerical derivation of training and generalization dynamics in linear networks.
result Grokking can occur in linear networks without reaching understanding, and its timing depends on various parameters.
The paper proposes a Gibbs sampler for neural network posterior sampling.
problem Sampling from the posterior of neural networks.
method Adding noise to activations and using a Gibbs sampler.
result The Gibbs sampler achieves similar performance to MCMC methods on real and synthetic data.
Neural networks' performance scales with data size, explained by data manifold dimensionality.
problem Understanding the scaling of neural network performance with the number of parameters.
method Explained by the intrinsic dimension of the data manifold, confirmed through teacher/student framework and various datasets.
result The scaling exponent α is approximately 4 divided by the intrinsic dimension d of the data manifold.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
Framework transfers limited steering angle data across multiple weather conditions.
problem Limited labeled data for diverse weather conditions in sensorimotor control.
method Teacher-student learning paradigm with image-to-image translation network.
result Framework generalizes well across multiple weather conditions using limited labels.
We analyze anomaly detection class imbalance using a solvable model.
problem Class imbalance hampers anomaly detection performance.
method We use an exact solution of the teacher-student perceptron model through replica theory.
result Optimal train imbalance is often different from 50%, influenced by intrinsic imbalance and data abundance.
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
This work measures how many training examples are needed for kernel methods to generalize well.
problem Estimating the number of training examples needed for kernel methods to generalize well.
method Empirical data and Teacher-Student framework for kernels, with analytical derivations for translation invariant kernels.
result The exponent β for kernel methods is found to be dependent on the smoothness and dimension of the training data.
Locality helps in learning from high-dimensional data.
problem Understanding how convolutional neural networks learn from high-dimensional data.
method Teacher-student framework for kernel regression with convolutional kernels.
result Locality is key to determining the learning curve exponent in high-dimensional data.
Paper presents a deep learning framework for classifying respiratory anomalies and lung diseases from sound recordings.
problem Classifying respiratory anomalies and lung diseases from respiratory sound recordings.
method The framework uses front-end feature extraction to transform sound into spectrograms, and a deep learning network to classify these features.
result The proposed deep learning system outperforms current state-of-the-art methods on the ICBHI benchmark dataset.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
Semi-supervised learning chain boosts histology image accuracy with minimal labeled data.
problem Expensive annotation of large histology image databases.
method Teacher-student knowledge distillation chain approach.
result Accuracy matches 100% labeled data with only 0.5% labeled data.
Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
Reduces multiclass and regression compression schemes to binary ones.
problem Developing efficient learning algorithms for multiclass and regression problems.
method Reduces sample compression schemes for binary classes to multiclass and regression settings.
result Establishes new compression schemes for multiclass and regression problems.