A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper generalizes teacher-student model for realistic data.
Study shows how task similarity affects forgetting in teacher-student setup.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
A new training method improves MLIPs for faster, lighter simulations.
Paper analyzes how neural networks learn from a teacher in a specific setting.
Analyzes Hessian spectrum for neural networks near optimal learning.
Randomly sampled interpolators achieve zero generalization error with enough data.
Study online learner attacks by manipulating labels, revealing critical thresholds.
High-dimensional data in many machine learning applications leads to computational and analytical complexities. Feature selection provides an effective way for solving these problems by removing irrelevant and redundant features, thus reducing model complexity and improving accuracy and generalization capability of the…
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
Superpixel-mix enhances reliability in semantic segmentation.
For real-world speech recognition applications, noise robustness is still a challenge. In this work, we adopt the teacher-student (T/S) learning technique using a parallel clean and noisy corpus for improving automatic speech recognition (ASR) performance under multimedia noise. On top of that, we apply a logits select…
Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the a…
Superposition accelerates training to a universal power-law exponent.
Approach for biosensor condition detection using domain adaptation.
Linear models can grok without understanding, improving generalization.
The paper proposes a Gibbs sampler for neural network posterior sampling.
Classifiers built with neural networks handle large-scale high dimensional data, such as facial images from computer vision, extremely well while traditional statistical methods often fail miserably. In this paper, we attempt to understand this empirical success in high dimensional classification by deriving the conver…
Neural networks' performance scales with data size, explained by data manifold dimensionality.
More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teac…
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
We analyze anomaly detection class imbalance using a solvable model.
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained…
Locality helps in learning from high-dimensional data.
Paper presents a deep learning framework for classifying respiratory anomalies and lung diseases from sound recordings.
The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student …
RAF model explains neural networks' dual rule learning and fact memorization.
A new method for training GNNs without a teacher model.
Semi-supervised learning chain boosts histology image accuracy with minimal labeled data.
Proves formula for reconstruction performance in generalized linear models.
We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the synaptic weights and neurons in the hidden layers in a simple feed-forward perc…
We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the r…
New algorithm outperforms existing ones by focusing on mastering rate.
This paper explores the use of adversarial examples in training speech recognition systems to increase robustness of deep neural network acoustic models. During training, the fast gradient sign method is used to generate adversarial examples augmenting the original training data. Different from conventional data augmen…
New PL method improves ASR accuracy without pseudo-labels.
A new method identifies a stable subnetwork in overparameterized student models.
Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework explicitly formul…
How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as where is the number of training examples and an exponent that depends on both data and algorithm. In this work we measure when applying kernel methods to real datasets. For …
Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the…
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a teacher-student setup, where one network, the student, is trained using stochastic gradient de…
Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers ne…
Meta Pseudo Labels boosts image classification accuracy to 90.2%.
Recent variants improve knowledge distillation performance.
Optimal task order improves continual learning performance.
New insights into neural network forgetting reveal a trade-off between node activation and re-use.