A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper generalizes teacher-student model for realistic data.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
Study shows how task similarity affects forgetting in teacher-student setup.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
A new training method improves MLIPs for faster, lighter simulations.
Paper analyzes how neural networks learn from a teacher in a specific setting.
Randomly sampled interpolators achieve zero generalization error with enough data.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
Study online learner attacks by manipulating labels, revealing critical thresholds.
High-dimensional data in many machine learning applications leads to computational and analytical complexities. Feature selection provides an effective way for solving these problems by removing irrelevant and redundant features, thus reducing model complexity and improving accuracy and generalization capability of the…
Analyzes Hessian spectrum for neural networks near optimal learning.
Approach for biosensor condition detection using domain adaptation.
For real-world speech recognition applications, noise robustness is still a challenge. In this work, we adopt the teacher-student (T/S) learning technique using a parallel clean and noisy corpus for improving automatic speech recognition (ASR) performance under multimedia noise. On top of that, we apply a logits select…
Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the a…
Linear models can grok without understanding, improving generalization.
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
Superposition accelerates training to a universal power-law exponent.
The paper proposes a Gibbs sampler for neural network posterior sampling.
More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teac…
Neural networks' performance scales with data size, explained by data manifold dimensionality.
Superpixel-mix enhances reliability in semantic segmentation.
The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student …
We analyze anomaly detection class imbalance using a solvable model.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
RAF model explains neural networks' dual rule learning and fact memorization.
A new method for training GNNs without a teacher model.
Paper presents a deep learning framework for classifying respiratory anomalies and lung diseases from sound recordings.
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained…
Classifiers built with neural networks handle large-scale high dimensional data, such as facial images from computer vision, extremely well while traditional statistical methods often fail miserably. In this paper, we attempt to understand this empirical success in high dimensional classification by deriving the conver…
Proves formula for reconstruction performance in generalized linear models.
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
Locality helps in learning from high-dimensional data.
New PL method improves ASR accuracy without pseudo-labels.
Semi-supervised learning chain boosts histology image accuracy with minimal labeled data.
This paper explores the use of adversarial examples in training speech recognition systems to increase robustness of deep neural network acoustic models. During training, the fast gradient sign method is used to generate adversarial examples augmenting the original training data. Different from conventional data augmen…
We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the r…
A new method identifies a stable subnetwork in overparameterized student models.
Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework explicitly formul…
We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the synaptic weights and neurons in the hidden layers in a simple feed-forward perc…
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a teacher-student setup, where one network, the student, is trained using stochastic gradient de…
Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers ne…
Recent variants improve knowledge distillation performance.
New algorithm outperforms existing ones by focusing on mastering rate.
This paper improves ASR performance by aligning frames more accurately.
Recent advances in deep learning have facilitated the demand of neural models for real applications. In practice, these applications often need to be deployed with limited resources while keeping high accuracy. This paper touches the core of neural models in NLP, word embeddings, and presents a new embedding distillati…
RKD improves model compression by distilling residual knowledge from a deep teacher model.
Optimal task order improves continual learning performance.
How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as where is the number of training examples and an exponent that depends on both data and algorithm. In this work we measure when applying kernel methods to real datasets. For …