Paper generalizes teacher-student model for realistic data.
problem Capturing learning curves for realistic datasets.
method Introduces a Gaussian covariate generalization of the teacher-student model.
result Generalized model captures learning curves for various realistic data sets.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
problem Understanding the excess risk of two-layer ReLU neural networks in teacher-student models.
method Investigated a two-phase training process for a student network, comparing it to kernel methods.
result The student network reaches near-global optimality and outperforms kernel methods in minimax optimal rate.
Randomly sampled interpolators achieve zero generalization error with enough data.
problem Understanding the high generalization ability of machine learning models.
method Algebraic geometry tools to prove zero generalization error for random interpolators.
result Generalization error of randomly sampled interpolators becomes zero once the number of training samples exceeds a geometric threshold.
A new training method improves MLIPs for faster, lighter simulations.
problem High computational and memory costs of complex MLIPs for large-scale MD simulations.
method Teacher-student training framework using latent atomic energy knowledge.
result Lightweight student MLIPs achieve faster MD speeds and comparable accuracy to teachers.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
Paper presents a deep learning framework for classifying respiratory anomalies and lung diseases from sound recordings.
problem Classifying respiratory anomalies and lung diseases from respiratory sound recordings.
method The framework uses front-end feature extraction to transform sound into spectrograms, and a deep learning network to classify these features.
result The proposed deep learning system outperforms current state-of-the-art methods on the ICBHI benchmark dataset.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
Superposition accelerates training to a universal power-law exponent.
problem Training dynamics in neural networks.
method Teacher-student framework and analytic theory.
result Superposition leads to a universal power-law exponent of ~1, independent of data and channel statistics.
We analyze anomaly detection class imbalance using a solvable model.
problem Class imbalance hampers anomaly detection performance.
method We use an exact solution of the teacher-student perceptron model through replica theory.
result Optimal train imbalance is often different from 50%, influenced by intrinsic imbalance and data abundance.
Classifiers built with neural networks handle large-scale high dimensional data, such as facial images from computer vision, extremely well while traditional statistical methods often fail miserably. In this paper, we attempt to understand this empirical success in high dimensional classification by deriving the conver…
Neural networks' performance scales with data size, explained by data manifold dimensionality.
problem Understanding the scaling of neural network performance with the number of parameters.
method Explained by the intrinsic dimension of the data manifold, confirmed through teacher/student framework and various datasets.
result The scaling exponent α is approximately 4 divided by the intrinsic dimension d of the data manifold.
Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework explicitly formul…
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained…
Locality helps in learning from high-dimensional data.
problem Understanding how convolutional neural networks learn from high-dimensional data.
method Teacher-student framework for kernel regression with convolutional kernels.
result Locality is key to determining the learning curve exponent in high-dimensional data.
Study online learner attacks by manipulating labels, revealing critical thresholds.
problem Adversarial attacks on online learners, especially label manipulation.
method Control-theoretical perspective, theoretical analysis, empirical study.
result Discontinuous transition in learner's accuracy occurs at a critical threshold.
High-dimensional data in many machine learning applications leads to computational and analytical complexities. Feature selection provides an effective way for solving these problems by removing irrelevant and redundant features, thus reducing model complexity and improving accuracy and generalization capability of the…
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
RAF model explains neural networks' dual rule learning and fact memorization.
problem Understanding how neural networks learn rules and memorize facts simultaneously.
method Introduces the Rules-and-Facts (RAF) model to bridge generalization and memorization.
result Characterizes conditions for simultaneous rule learning and fact memorization in neural networks.
Superpixel-mix enhances reliability in semantic segmentation.
problem Improving reliability in real-world semantic segmentation.
method Superpixel-mix, a new data augmentation method with teacher-student consistency training.
result Superpixel-mix achieves state-of-the-art results in semi-supervised semantic segmentation.
For real-world speech recognition applications, noise robustness is still a challenge. In this work, we adopt the teacher-student (T/S) learning technique using a parallel clean and noisy corpus for improving automatic speech recognition (ASR) performance under multimedia noise. On top of that, we apply a logits select…
Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the a…
Approach for biosensor condition detection using domain adaptation.
problem Limited clinical data for biosensor condition detection.
method Teacher-student adaptation of deep learning models.
result Significantly outperforms other methods on synthetic and real-world datasets.
Linear models can grok without understanding, improving generalization.
problem Understanding the phenomenon of grokking in linear models.
method Analytical and numerical derivation of training and generalization dynamics in linear networks.
result Grokking can occur in linear networks without reaching understanding, and its timing depends on various parameters.
The paper proposes a Gibbs sampler for neural network posterior sampling.
problem Sampling from the posterior of neural networks.
method Adding noise to activations and using a Gibbs sampler.
result The Gibbs sampler achieves similar performance to MCMC methods on real and synthetic data.
Study characterizes training and test risks for MAP regression with Gaussian priors.
problem Understanding high-dimensional behavior of regularized linear regression with informative priors.
method Maximum a posteriori (MAP) regression with Gaussian priors, using random matrix theory.
result Closed-form risk formulas reveal the bias-variance-prior tradeoff and explain double descent.
More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teac…
New PL method improves ASR accuracy without pseudo-labels.
problem Improving ASR accuracy with limited labeled data.
method End-to-end continuous pseudo-labeling with soft-labels.
result Soft-labels can lead to model collapse, but regularization can mitigate this.
A new method identifies a stable subnetwork in overparameterized student models.
problem Identifying a stable subnetwork in overparameterized student models.
method Spectral representation of linear transfer of information, focusing on eigenvalues and eigenvectors.
result A stable student substructure is isolated that mirrors the true complexity of the teacher.
The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student …
Optimal task order improves continual learning performance.
problem Challenges in neural networks learning multiple tasks in sequence.
method Linear teacher-student model with latent factors, derived analytical expression.
result Two principles for optimal task order: least representative first and dissimilar adjacent tasks.
How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as n−β where n is the number of training examples and β an exponent that depends on both data and algorithm. In this work we measure β when applying kernel methods to real datasets. For …
Semi-supervised learning chain boosts histology image accuracy with minimal labeled data.
problem Expensive annotation of large histology image databases.
method Teacher-student knowledge distillation chain approach.
result Accuracy matches 100% labeled data with only 0.5% labeled data.
New insights into neural network forgetting reveal a trade-off between node activation and re-use.
problem Challenges in maintaining performance on old tasks while learning new ones.
method Theoretical analysis of synthetic and real data setups, focusing on node activation vs re-use.
result Worst forgetting occurs in an intermediate similarity regime between learned tasks.
This paper improves ASR performance by aligning frames more accurately.
problem Disagreement between teacher-student models in frame-level alignment.
method Introduces self-knowledge distillation (SKD) to guide frame-level alignment.
result Improves both resource efficiency and performance in ASR.
ASGN uses active semi-supervised learning to predict molecular properties efficiently.
problem Predicting molecular properties with scarce labeled data and high computational cost.
method ASGN combines a teacher-student framework with active learning to handle joint representation and property learning.
result ASGN achieves remarkable performance in property prediction on public datasets.
The paper proposes a machine learning framework for portfolio optimization with limited data.
problem Low data environments and regime uncertainty in portfolio optimization.
method A teacher-student learning pipeline with CVaR optimizer generating supervisory labels and neural models trained on real and synthetic data.
result Student models can match or outperform the CVaR teacher and achieve improved robustness under regime shifts.
Proves formula for reconstruction performance in generalized linear models.
problem Analyzing reconstruction performance in generalized linear models with arbitrary bounded spectrum.
method Message passing algorithms and dynamical system stability analysis.
result Analytical formula confirms replica method conjecture for convex models.
The ability of deep learning models to generalize well across different scenarios depends primarily on the quality and quantity of annotated data. Labeling large amounts of data for all possible scenarios that a model may encounter would not be feasible; if even possible. We propose a framework to deal with limited lab…
Recent advances in deep learning have facilitated the demand of neural models for real applications. In practice, these applications often need to be deployed with limited resources while keeping high accuracy. This paper touches the core of neural models in NLP, word embeddings, and presents a new embedding distillati…
We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and th…
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the synaptic weights and neurons in the hidden layers in a simple feed-forward perc…