Student-teacher learning improves generalization with noisy inputs.
problem Transfer knowledge from clean inputs to noisy inputs.
method Analyzes student-teacher learning using deep linear networks and experiments with nonlinear networks.
result Three factors are vital for success: zero training loss, teacher knowledge, and feature decomposition.
Proposes RaT to mitigate bias in student-teacher estimation.
problem Systematic bias in teacher's predictions propagates to student model.
method Uses teacher to estimate residuals in student's predictions.
result RaT method reduces teacher bias effect and achieves optimal rate.
Self-training with noisy student-teacher boosts keyword spotting accuracy.
problem Robust keyword spotting in challenging conditions.
method Aggressive data augmentation and self-training with noisy student-teacher approach.
result Significant accuracy improvement in difficult conditions, up to 60%.
This work proposes a student-teacher network for predicting hospital admission locations.
problem Accurate prediction of hospital admission locations to optimize resource allocation.
method Reinforcement learning approach where a teacher network selects data batches for a student network.
result The approach outperforms state-of-the-art methods on tabular data and image recognition.
A new method detects concept drift without true labels.
problem Detecting concept drift in unsupervised settings.
method Student-teacher learning paradigm for drift detection.
result The method outperforms state-of-the-art approaches in experiments.
KD can lead to student-teacher deviations that improve performance.
problem KD can lead to student-teacher deviations that may outperform the teacher.
method Characterized and explained the nature of student-teacher deviations through experiments and theory.
result KD can lead to improved generalization by exaggerating the implicit bias of gradient descent.
Framework models supervised learning in non-stationary data.
problem Non-stationary data in supervised learning.
method Statistical physics methods applied to LVQ and neural networks.
result LVQ and ReLU have different sensitivity to concept drift.
LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.
problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.
method Generalized student-teacher setting with i.i.d. samples and online gradient descent.
result Gradient descent converges to the teacher model in dkO(1) iterations under certain conditions. Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.
problem Understanding the effectiveness of random feature regression with nearly orthogonal data.
method Investigates RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime.
result Shows high-probability non-asymptotic concentration results for RFRR's training, cross-validation, and generalization errors.
This is a report of our lessons learned building acoustic models from 1 Million hours of unlabeled speech, while labeled speech is restricted to 7,000 hours. We employ student/teacher training on unlabeled data, helping scale out target generation in comparison to confidence model based methods, which require a decoder…
Large scale machine learning (ML) systems such as the Alexa automatic speech recognition (ASR) system continue to improve with increasing amounts of manually transcribed training data. Instead of scaling manual transcription to impractical levels, we utilize semi-supervised learning (SSL) to learn acoustic models (AM) …
Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner, where knowledge gained from previous tasks is retained and used to aid future learning over the lifetime of the learner. It is essential towards the development of intelligent machines that can adapt to their surroundings. I…
We analyze the dynamics of an algorithm for approximate inference with large Gaussian latent variable models in a student-teacher scenario. To model nontrivial dependencies between the latent variables, we assume random covariance matrices drawn from rotation invariant ensembles. For the case of perfect data-model matc…
Machine learning maps knots to embeddings, revealing topological invariants.
problem Learning topological invariance in knot theory.
method Contrastive and generative machine learning techniques, auto-regressive decoder Transformer network.
result Neural networks can map different knots to the same point in an embedding vector space.
We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…
Although reinforcement learning (RL) can provide reliable solutions in many settings, practitioners are often wary of the discrepancies between the RL solution and their status quo procedures. Therefore, they may be reluctant to adapt to the novel way of executing tasks proposed by RL. On the other hand, many real-worl…
Study analyzes learning dynamics in nonlinear perceptrons using stochastic-process approach.
problem Understanding the roles of nonlinearity and input-data distribution in neural network learning.
method Stochastic-process approach to derive flow equations for learning dynamics in nonlinear perceptrons.
result Input-data noise affects learning speed differently under supervised and reinforcement learning.
BootsTAP uses real-world data to improve TAP tracking performance.
problem Lack of large-scale groundtruth training data for TAP.
method Bootstrapped training with a self-supervised student-teacher setup.
result State-of-the-art performance on TAP-Vid benchmarks.
Overcoming the visual barrier and developing "see-through vision" has been one of mankind's long-standing dreams. Unlike visible light, Radio Frequency (RF) signals penetrate opaque obstructions and reflect highly off humans. This paper establishes a deep-learning model that can be trained to reconstruct continuous vid…
A new gradient flow for MMD with closed-form implementation.
problem Existing gradient flows either lack tractable numerical implementation or require strong assumptions.
method Introduces a (de)-regularized Maximum Mean Discrepancy (DrMMD) and its gradient flow.
result Guarantees near-global convergence for a broad class of targets in both continuous and discrete time.
This work analyzes neural scaling laws using power-law data spectra and derives analytical expressions for generalization error.
problem Understanding how neural network performance scales with key factors like data size and model complexity.
method Statistical mechanics techniques applied to one-pass stochastic gradient descent in a student-teacher framework.
result Derivation of analytical expressions for generalization error under power-law data spectra and identification of conditions for power-law scaling.
User-generated reviews can be decomposed into fine-grained segments (e.g., sentences, clauses), each evaluating a different aspect of the principal entity (e.g., price, quality, appearance). Automatically detecting these aspects can be useful for both users and downstream opinion mining applications. Current supervised…
One gradient step improves neural network feature learning by aligning weights with the teacher model.
problem Improving feature learning in neural networks through gradient descent.
method First gradient descent step on the first-layer parameters of a two-layer neural network.
result The first gradient update contains a rank-1 spike, leading to alignment with the teacher model.
Grokking occurs when neural networks transition from lazy to rich training dynamics, fitting initial features before generalizing.
problem Understanding why neural networks exhibit early train loss decrease without corresponding test loss improvement.
method Analyzing vanilla gradient descent on polynomial regression with a two-layer neural network, identifying sufficient statistics for test loss.
result Grokking arises when a network first attempts to fit a kernel regression solution with initial features, followed by late-time feature learning.
One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
PETAL adapts models to changing target domains over time.
problem Lifelong test-time adaptation in changing target domains.
method Probabilistic framework with student-teacher model and data-driven parameter restoration.
result PETAL achieves better results than state-of-the-art for online lifelong test-time adaptation.
New metric improves latent dynamics inference from neural data.
problem Limitations of co-smoothing in predicting latent dynamics.
method Few-shot co-smoothing to assess latent dynamics.
result High co-smoothing models often have extraneous dynamics, which few-shot co-smoothing detects.
Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model. Hint training (i.e., FitNets) extends KD by regressing a student model's intermediate representation to a teacher mo…
Meta-learning adapts models for unseen tasks across AI, robotics, and NLP.
problem Adapting models to unseen tasks efficiently and accurately.
method Black-box, metric-based, layered, and Bayesian approaches.
result Meta-learning enhances model generalization and adaptation to unseen tasks.
Meta-learning improves neural networks by adapting learning algorithms.
problem Conventional AI approaches solve tasks from scratch, but meta-learning aims to improve the learning algorithm.
method Meta-learning adapts a learning algorithm based on multiple learning episodes.
result Meta-learning can tackle deep learning challenges like data and computation bottlenecks.
Survey explores how transfer learning improves deep reinforcement learning.
problem Challenges in reinforcement learning efficiency and effectiveness.
method Categorizes and analyzes transfer learning approaches.
result Transfer learning enhances reinforcement learning performance.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
Poisson learning doesn't solve graph semi-supervised learning issues.
problem Global information loss in graph-based semi-supervised learning.
method Poisson learning is Laplace regularization with thresholding.
result Poisson learning cannot overcome the global information loss problem.
New unsupervised learning technique learns independent kernels for better machine learning tasks.
problem Improving unsupervised representation learning for machine learning tasks.
method Stacking convolutional transforms using alternating proximal minimization scheme.
result DCTL outperforms shallow version CTL on benchmark datasets.
Meta-learning helps models learn quickly from few samples.
problem Deep learning requires many samples, which are hard to get.
method Meta-learning optimizes models to adapt quickly to new tasks.
result Meta-learning can improve model efficiency and adaptability.
New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
Deep reinforcement learning finds optimal learning policies for adaptive systems.
problem Finding individualized learning plans for learners with unknown latent traits.
method Formulated as a Markov decision process, applied deep Q-learning with a transition model estimator.
result The algorithm efficiently discovers optimal learning policies with small data sets.
Unified framework explains all types of learning, including brain.
problem Lack of clear explanation for deep learning success.
method Constructing a learning principle that equates all learning to probability estimation.
result Unified understanding of learning across different fields.
Cyclical learning rates improve DRL performance without manual tuning.
problem Manual hyperparameter tuning in DRL is time-consuming and error-prone.
method Proposes cyclical learning rates for DRL problems.
result Cyclical learning achieves similar or better results than fixed learning rates.
Study batch reinforcement learning methods for personalized medical treatments.
problem Batch reinforcement learning for personalized medical treatments.
method Direct policy learning and model-based learning approaches.
result Model-based learning is impossible with finite model classes but feasible with relaxed conditions.
A new meta-meta classification method tackles few-shot learning tasks.
problem Learning with limited data in small-data settings.
method Designing an ensemble of learners for a large set of problems, then learning how to combine them for a new problem.
result Meta-meta classification outperforms traditional meta-learning and ensembling approaches in one-shot learning tasks.
The paper proposes a learning algorithm that improves adaptability and generalization.
problem Improving adaptability and generalization in learning models.
method Learning to meta-learn by meta-finetuning on related tasks before adapting to specific tasks.
result Learning to meta-learn improves adaptability and generalization across various tasks.
A common strategy in modern learning systems is to learn a representation that is useful for many tasks, a.k.a. representation learning. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We formulate representation learning …
This paper surveys meta-learning, online, and continual learning.
problem Combining and understanding meta-learning, online, and continual learning.
method Organizing various problem settings using consistent terminology and formal descriptions.
result Fosters further advancements in meta-learning, online, and continual learning.
Relational logistic regression (RLR) is a representation of conditional probability in terms of weighted formulae for modelling multi-relational data. In this paper, we develop a learning algorithm for RLR models. Learning an RLR model from data consists of two steps: 1- learning the set of formulae to be used in the m…
Survey on curriculum learning for reinforcement learning.
problem Efficiently learning complex tasks in reinforcement learning.
method Framework for curriculum learning in reinforcement learning.
result Survey and classification of existing CL methods.
Contrastive learning works well with redundant data views.
problem Improving unsupervised learning with redundant data views.
method Theoretical analysis of contrastive learning in a multi-view setting.
result Linear functions of learned representations are nearly optimal on prediction tasks when views are redundant.
Paper analyzes iterative learning for concept classes and learns half-spaces.
problem Learning concept classes efficiently with iterative learners.
method Analyzes various settings of iterative learning and provides a constructive algorithm for half-spaces.
result Constructive iterative algorithm for learning half-spaces from informant.