Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,878 papers · 148 categories
Trend · papers per month
48 results for “student network”
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
problem Overwhelming size of Deep Neural Networks (DNNs).
method Single teacher with multiple student networks, transferring knowledge to each student.
result The combined knowledge of the class of students achieves better performance and reduces parameters.
Student-teacher learning improves generalization with noisy inputs.
problem Transfer knowledge from clean inputs to noisy inputs.
method Analyzes student-teacher learning using deep linear networks and experiments with nonlinear networks.
result Three factors are vital for success: zero training loss, teacher knowledge, and feature decomposition.
Predicts student performance in interactive online question pools using GNNs.
problem Predicting student performance in interactive online question pools with evolving knowledge.
method Proposes R^2GCN, a GNN model for heterogeneous networks to predict student performance.
result Achieves higher accuracy in student performance prediction than traditional methods.
To reduce the large computation and storage cost of a deep convolutional neural network, the knowledge distillation based methods have pioneered to transfer the generalization ability of a large (teacher) deep network to a light-weight (student) network. However, these methods mostly focus on transferring the probabili…
Two-layer ReLU networks outperform kernel methods in teacher-student settings.
problem Understanding the excess risk of two-layer ReLU neural networks in teacher-student models.
method Investigated a two-phase training process for a student network, comparing it to kernel methods.
result The student network reaches near-global optimality and outperforms kernel methods in minimax optimal rate.
Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teache…
Paper improves neural network models for MOOC student course prediction.
problem Improving neural network-based predictive methods for MOOC student course trajectory modeling.
method Enhanced LSTM networks and Transformer architecture applied to MOOC student data.
result Improved accuracy in predicting MOOC student course trajectories.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
Study predicts academic achievement using students' support networks.
problem Predicting academic achievement in college students.
method Decision tree and random forest algorithms applied to Ties data.
result Different types of support are important for different demographics and genders.
Student network specializes teacher nodes in deep ReLU networks.
problem Training deep ReLU networks with finite width and input dimension.
method Stochastic Gradient Descent (SGD) on over-realized student network trained from teacher network output.
result Each teacher node is specialized by at least one student node at the lowest layer under mild conditions.
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
This work proposes a student-teacher network for predicting hospital admission locations.
problem Accurate prediction of hospital admission locations to optimize resource allocation.
method Reinforcement learning approach where a teacher network selects data batches for a student network.
result The approach outperforms state-of-the-art methods on tabular data and image recognition.
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robu…
Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teac…
On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingstat.ML
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
Enhances student diversity in collaborative learning.
problem Student homogenization in large groups.
method Random routing, diverse feature sets, and random subgroup imitation.
result Significantly outperforms state-of-the-art approaches.
A new system recommends specific knowledge concepts in MOOCs based on student interests.
problem MOOCs recommend courses but ignore specific knowledge concepts interests.
method End-to-end graph neural network (ACKRec) that combines content and context information.
result ACKRec effectively recommends knowledge concepts to MOOC students.
Novel framework learns efficient student models from teacher networks.
problem Model capacity gap between teacher and student networks.
method Neural architecture search and oracle knowledge distillation.
result Searched student models often outperform teacher models.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupstat.ML
Study on SGD dynamics in neural networks, revealing generalisation patterns.
problem Understanding generalisation in over-parameterised neural networks.
method Analysis of SGD dynamics in a teacher-student setup using differential equations.
result Network size affects generalisation error differently depending on training layers and activation functions.
Distilled models often fail to match teacher models, despite improving generalization.
problem The discrepancy between teacher and student predictive distributions remains large.
method Investigated the optimization difficulties and dataset details affecting student performance.
result Optimizing for matching the teacher does not always lead to better generalization.
A method distills GANs for mobile devices, reducing computation and storage.
problem Heavy computation and storage cost of GANs on mobile devices.
method Knowledge distillation to train a smaller generator with inherited information from a larger teacher generator, including a discriminator.
result Portable GAN models with strong performance achieved.
The task of accelerating large neural networks on general purpose hardware has, in recent years, prompted the use of channel pruning to reduce network size. However, the efficacy of pruning based approaches has since been called into question. In this paper, we turn to distillation for model compression---specifically,…
Knowledge transfer speeds up neural classifier training.
problem Lack of theoretical analysis of knowledge transfer in neural networks.
method Regularization of fit between teacher and student networks using privileged information.
result Wide two-layer networks can interpolate between privileged information and data, improving generalization.
Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model. Hint training (i.e., FitNets) extends KD by regressing a student model's intermediate representation to a teacher mo…
NeuralCD diagnoses student proficiency in exercises using neural networks.
problem Capturing complex student-exercise interactions for accurate cognitive diagnosis.
method Proposes NeuralCD framework using neural networks to learn complex interactions between students and exercises, incorporating monotonicity for interpretability.
result Demonstrates effectiveness of NeuralCD framework on real-world datasets, achieving both accuracy and interpretability.
A new method identifies a stable subnetwork in overparameterized student models.
problem Identifying a stable subnetwork in overparameterized student models.
method Spectral representation of linear transfer of information, focusing on eigenvalues and eigenvectors.
result A stable student substructure is isolated that mirrors the true complexity of the teacher.
Study analyzes catastrophic forgetting in continual learning using teacher-student networks.
problem Catastrophic forgetting in continuously learning systems.
method Teacher-student learning framework, similarity of input distributions and target functions.
result Network can avoid catastrophic forgetting with small input distribution similarity and large target function similarity.
Deep neural network compression techniques such as pruning and weight tensor decomposition usually require fine-tuning to recover the prediction accuracy when the compression ratio is high. However, conventional fine-tuning suffers from the requirement of a large training set and the time-consuming training procedure. …
A new method for student-initiated action advice using novelty detection.
problem Exploration and sample inefficiency in RL, especially with teacher absence.
method Random Network Distillation (RND) to measure advice novelty, updates only for advised states.
result Significant performance improvement over state-of-the-art methods, especially in challenging scenarios.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Analyzes Hessian spectrum for neural networks near optimal learning.
problem Understanding learning dynamics near optimal points in neural networks.
method Characterizes Hessian eigenspectrum for teacher-student problems, using analytical and numerical methods.
result The rank of the Hessian matrix determines effective number of parameters for non-linear networks.
DeepMimic trains neural networks using mostly unlabeled data.
problem Insufficient labeled data for deep learning.
method Mentor-Student approach with unlabeled data.
result Student model achieves mentor's performance without labeled data.
Optimal transport metric transfers deep network representations efficiently.
problem Efficiently transfer deep network representations for new tasks.
method Use optimal transport to quantify representation similarity and regularize student network.
result Optimal transport distance promotes similarity between teacher and student representations.
Study shows how task similarity affects forgetting in teacher-student setup.
problem Catastrophic forgetting in continual learning.
method Extended teacher-student setup to multiple teachers, analyzing similarity between tasks.
result Task similarity, whether at readouts or features, influences forgetting and transfer.
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a teacher-student setup, where one network, the student, is trained using stochastic gradient de…
SiamJEPA uses Siamese student encoders to improve JEPA-based representation learning.
problem Improving self-supervised representation learning in JEPA models.
method Proposes SiamJEPA with masked Siamese student encoders and EMA teacher network.
result Siamese student encoders improve representation separability and learning speed.
In this paper we do the first large scale analysis of writing style development among Danish high school students. More than 10K students with more than 100K essays are analyzed. Writing style itself is often studied in the natural language processing community, but usually with the goal of verifying authorship, assess…
Bayesian Dark Knowledge is a method for compressing the posterior predictive distribution of a neural network model into a more compact form. Specifically, the method attempts to compress a Monte Carlo approximation to the parameter posterior into a single network representing the posterior predictive distribution. Fur…
Study predicts social relationships using triadic influence from social networks.
problem Difficulty in quantifying social relationships and their dynamics.
method Real social networks of 13 schools, neural networks, high-dimensional embedding.
result Triadic influence achieves highest accuracy in predicting student relationships.
Researchers study how teachers' advising relationships influence their perceptions of satisfaction and students, not policy influence.
problem Understanding the relationship between teachers' advising relationships and their perceptions of satisfaction and students.
method Proposed a novel joint model of network and item responses (JNIRM) with correlated latent variables.
result Teachers' advising relationships contribute more to satisfaction and students than to influence over educational policies.
Expanding neural networks improves their learning from noisy data.
problem Improving neural network performance in noisy conditions.
method Sparse expansion of neural network inputs, followed by pruning, enhances generalization.
result Sparse expansion of neural networks improves generalization performance, even after pruning.
KD can lead to student-teacher deviations that improve performance.
problem KD can lead to student-teacher deviations that may outperform the teacher.
method Characterized and explained the nature of student-teacher deviations through experiments and theory.
result KD can lead to improved generalization by exaggerating the implicit bias of gradient descent.
Deep neural networks forecast financial return distributions accurately.
problem Forecasting probability distributions of financial returns.
method Used 1D CNN and LSTM architectures with custom loss functions to optimize distribution parameters.
result LSTM with skewed Student's t distribution outperformed classical models in multiple evaluation metrics.
Deep learning based knowledge tracing model has been shown to outperform traditional knowledge tracing model without the need for human-engineered features, yet its parameters and representations have long been criticized for not being explainable. In this paper, we propose Deep-IRT which is a synthesis of the item res…
Generative Adversarial Networks (GANs) have a great performance in image generation, but they need a large scale of data to train the entire framework, and often result in nonsensical results. We propose a new method referring to conditional GAN, which equipments the latent noise with mixture of Student's t-distributio…
Meta Pseudo Labelscs.LG
Meta Pseudo Labels boosts image classification accuracy to 90.2%.
problem Improving semi-supervised learning for image classification.
method Adapts a teacher network to generate better pseudo labels through student feedback.
result Achieves a new state-of-the-art top-1 accuracy of 90.2% on ImageNet.
Paper generalizes teacher-student model for realistic data.
problem Capturing learning curves for realistic datasets.
method Introduces a Gaussian covariate generalization of the teacher-student model.
result Generalized model captures learning curves for various realistic data sets.