Paper generalizes teacher-student model for realistic data.
problem Capturing learning curves for realistic datasets.
method Introduces a Gaussian covariate generalization of the teacher-student model.
result Generalized model captures learning curves for various realistic data sets.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
problem Understanding the excess risk of two-layer ReLU neural networks in teacher-student models.
method Investigated a two-phase training process for a student network, comparing it to kernel methods.
result The student network reaches near-global optimality and outperforms kernel methods in minimax optimal rate.
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
Randomly sampled interpolators achieve zero generalization error with enough data.
problem Understanding the high generalization ability of machine learning models.
method Algebraic geometry tools to prove zero generalization error for random interpolators.
result Generalization error of randomly sampled interpolators becomes zero once the number of training samples exceeds a geometric threshold.
This work improves ASR noise robustness using parallel data and T/S learning.
problem Noise robustness in automatic speech recognition.
method Teacher-student learning with parallel clean and noisy data, logits selection.
result Best student model yields significant WER reductions in noisy conditions.
A novel feature selection method using a teacher-student network.
problem Feature selection for high-dimensional data in machine learning.
method Teacher-student network approach for feature selection.
result The proposed TSFS method outperforms existing methods in classification and clustering.
Study on SGD dynamics in neural networks, revealing generalisation patterns.
problem Understanding generalisation in over-parameterised neural networks.
method Analysis of SGD dynamics in a teacher-student setup using differential equations.
result Network size affects generalisation error differently depending on training layers and activation functions.
Deep neural networks outperform traditional methods in high-dimensional classification.
problem Understanding the empirical success of deep neural networks in high-dimensional classification.
method Proposed a teacher-student framework with Bayes classifier as ReLU neural networks, derived convergence rates for 0-1 and hinge losses.
result Sharp rate of convergence for classifiers trained using 0-1 or hinge loss, with Od(n−2/3) or Od(n−1) under separable data distribution. Study online learner attacks by manipulating labels, revealing critical thresholds.
problem Adversarial attacks on online learners, especially label manipulation.
method Control-theoretical perspective, theoretical analysis, empirical study.
result Discontinuous transition in learner's accuracy occurs at a critical threshold.
Proposes a new theoretical framework for deep locally connected ReLU networks.
problem Understanding theoretical properties of deep locally connected networks.
method Teacher-student setting, explicit formulation of data distribution, disentangled representations, compatibility with regularization techniques.
result Theoretical framework facilitates analysis of practical issues like overfitting and generalization.
Linear generalisation error with network size in neural networks.
problem Understanding generalisation in over-parameterised neural networks.
method Teacher-student setup with SGD dynamics captured by differential equations.
result Generalisation error increases linearly with network size.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
The paper proposes a Gibbs sampler for neural network posterior sampling.
problem Sampling from the posterior of neural networks.
method Adding noise to activations and using a Gibbs sampler.
result The Gibbs sampler achieves similar performance to MCMC methods on real and synthetic data.
A new training method improves MLIPs for faster, lighter simulations.
problem High computational and memory costs of complex MLIPs for large-scale MD simulations.
method Teacher-student training framework using latent atomic energy knowledge.
result Lightweight student MLIPs achieve faster MD speeds and comparable accuracy to teachers.
Linear models can grok without understanding, improving generalization.
problem Understanding the phenomenon of grokking in linear models.
method Analytical and numerical derivation of training and generalization dynamics in linear networks.
result Grokking can occur in linear networks without reaching understanding, and its timing depends on various parameters.
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
New approach uses synthetic labels to train models on scarce annotated data for surgical phase recognition.
problem Learning surgical phase recognition from limited annotated data.
method Teacher/Student approach with a CNN-biLSTM-CRF teacher generating synthetic labels for a CNN-LSTM student.
result Improved surgical phase recognition performance with fewer annotated videos.
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
Locality helps in learning from high-dimensional data.
problem Understanding how convolutional neural networks learn from high-dimensional data.
method Teacher-student framework for kernel regression with convolutional kernels.
result Locality is key to determining the learning curve exponent in high-dimensional data.
GAN-TSC improves student model accuracy by augmenting with synthetic data.
problem Suboptimal compression when fresh data is unavailable for TSC.
method Augment compression dataset with synthetic data from a GAN.
result Significant improvement in student accuracy for expensive models.
This paper proposes a method to embed teacher knowledge into a student network without increasing parameters.
problem The need for portable neural networks on mobile devices with limited resources.
method Feature embedding approach to distill knowledge from a teacher network to a student network without introducing new parameters.
result The proposed method maintains the performance of the teacher network while significantly reducing computational and storage complexity.
Conditional T/S learning improves student model performance by selectively learning from teacher or ground truth.
problem Teacher's occasional wrong guidance leads to suboptimal student model performance.
method Proposes a conditional T/S learning scheme where the student selectively chooses between teacher and ground truth based on teacher correctness.
result The conditional learning achieves significant performance improvements over traditional T/S learning.
Superpixel-mix enhances reliability in semantic segmentation.
problem Improving reliability in real-world semantic segmentation.
method Superpixel-mix, a new data augmentation method with teacher-student consistency training.
result Superpixel-mix achieves state-of-the-art results in semi-supervised semantic segmentation.
RAF model explains neural networks' dual rule learning and fact memorization.
problem Understanding how neural networks learn rules and memorize facts simultaneously.
method Introduces the Rules-and-Facts (RAF) model to bridge generalization and memorization.
result Characterizes conditions for simultaneous rule learning and fact memorization in neural networks.
Semi-supervised learning chain boosts histology image accuracy with minimal labeled data.
problem Expensive annotation of large histology image databases.
method Teacher-student knowledge distillation chain approach.
result Accuracy matches 100% labeled data with only 0.5% labeled data.
Superposition accelerates training to a universal power-law exponent.
problem Training dynamics in neural networks.
method Teacher-student framework and analytic theory.
result Superposition leads to a universal power-law exponent of ~1, independent of data and channel statistics.
Approach for biosensor condition detection using domain adaptation.
problem Limited clinical data for biosensor condition detection.
method Teacher-student adaptation of deep learning models.
result Significantly outperforms other methods on synthetic and real-world datasets.
Proves formula for reconstruction performance in generalized linear models.
problem Analyzing reconstruction performance in generalized linear models with arbitrary bounded spectrum.
method Message passing algorithms and dynamical system stability analysis.
result Analytical formula confirms replica method conjecture for convex models.
Deep neural networks undergo hierarchical free-energy landscape transitions with increasing data size.
problem Understanding the design space and dynamics of deep neural networks.
method Statistical mechanical approach based on replica method.
result Hierarchical free-energy landscape transitions with ultrametricity, leading to simpler configurations in deeper layers.
Neural networks' performance scales with data size, explained by data manifold dimensionality.
problem Understanding the scaling of neural network performance with the number of parameters.
method Explained by the intrinsic dimension of the data manifold, confirmed through teacher/student framework and various datasets.
result The scaling exponent α is approximately 4 divided by the intrinsic dimension d of the data manifold.
We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the r…
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
Framework transfers limited steering angle data across multiple weather conditions.
problem Limited labeled data for diverse weather conditions in sensorimotor control.
method Teacher-student learning paradigm with image-to-image translation network.
result Framework generalizes well across multiple weather conditions using limited labels.
A method for semi-supervised sound event detection using teacher-student learning.
problem Weakly-labeled data in sound event detection.
method Guided Learning with a teacher model for audio tagging and a student model for boundary detection.
result The method improves boundary detection performance using unlabeled data.
We analyze anomaly detection class imbalance using a solvable model.
problem Class imbalance hampers anomaly detection performance.
method We use an exact solution of the teacher-student perceptron model through replica theory.
result Optimal train imbalance is often different from 50%, influenced by intrinsic imbalance and data abundance.
This work measures how many training examples are needed for kernel methods to generalize well.
problem Estimating the number of training examples needed for kernel methods to generalize well.
method Empirical data and Teacher-Student framework for kernels, with analytical derivations for translation invariant kernels.
result The exponent β for kernel methods is found to be dependent on the smoothness and dimension of the training data.
Paper presents a deep learning framework for classifying respiratory anomalies and lung diseases from sound recordings.
problem Classifying respiratory anomalies and lung diseases from respiratory sound recordings.
method The framework uses front-end feature extraction to transform sound into spectrograms, and a deep learning network to classify these features.
result The proposed deep learning system outperforms current state-of-the-art methods on the ICBHI benchmark dataset.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers ne…
SGD tends to favor simpler subnetworks, improving generalization.
problem SGD's tendency to favor simpler subnetworks over complex ones.
method Identifying invariant sets and analyzing SGD's behavior around them.
result SGD collapses networks to simpler subnetworks, improving generalization.
The paper extends entropy maximization to multiscale settings and applies it to neural networks.
problem Achieving optimal risk bounds in neural networks using multiscale entropy.
method Generalizing maximum entropy to multiscale settings and applying it to neural networks.
result The multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior in a teacher-student scenario.
New algorithm outperforms existing ones by focusing on mastering rate.
problem Low sample efficiency in learning progress-based algorithms.
method Proposes a new algorithm based on mastering rate.
result Significantly outperforms learning progress-based algorithms.
This paper explores the use of adversarial examples in training speech recognition systems to increase robustness of deep neural network acoustic models. During training, the fast gradient sign method is used to generate adversarial examples augmenting the original training data. Different from conventional data augmen…
New PL method improves ASR accuracy without pseudo-labels.
problem Improving ASR accuracy with limited labeled data.
method End-to-end continuous pseudo-labeling with soft-labels.
result Soft-labels can lead to model collapse, but regularization can mitigate this.
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.