Real-time policy distillation speeds up and improves reinforcement learning.
problem Slow and inefficient policy distillation in reinforcement learning.
method Simultaneous training and distillation of a teacher and student model.
result Significantly reduced distillation time and improved small model performance.
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the…
Paper introduces a simulator-free approach to reinforcement learning policy distillation.
problem Learning multiplicity of cases corresponding to a given action in reinforcement learning.
method Generative adversarial approach to find multiple exemplars for each output class.
result Improves over state-of-the-art on data-free learning of student networks.
We focus on the problem of teaching a robot to solve tasks presented sequentially, i.e., in a continual learning scenario. The robot should be able to solve all tasks it has encountered, without forgetting past tasks. We provide preliminary work on applying Reinforcement Learning to such setting, on 2D navigation tasks…
ESPD improves learning efficiency in sparse reward reinforcement learning.
problem Sparse reward reinforcement learning challenges.
method Evolutionary Stochastic Policy Distillation (ESPD) based on drifted random walk insight.
result High learning efficiency demonstrated in MuJoCo robotics control suite experiments.
Paper proposes a novel policy distillation method for better order execution in noisy markets.
problem Effective order execution in noisy and imperfect market conditions.
method Policy distillation method to guide reinforcement learning towards optimal trading strategies.
result Significant improvements over various baselines in order execution.
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4imes faster per training step. Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each…
In multi-task reinforcement learning there are two main challenges: at training time, the ability to learn different policies with a single model; at test time, inferring which of those policies applying without an external signal. In the case of continual reinforcement learning a third challenge arises: learning tasks…
Deep reinforcement learning (DRL) is a booming area of artificial intelligence. Many practical applications of DRL naturally involve more than one collaborative learners, making it important to study DRL in a multi-agent context. Previous research showed that effective learning in complex multi-agent systems demands fo…
New approach makes deep reinforcement learning robust without assuming adversary knowledge.
problem Deep reinforcement learning policies are vulnerable to state observation perturbations.
method Proposes an adversary agnostic robust DRL paradigm using policy distillation with two terms: prescription gap maximization and Jacobian regularization.
result Boosts adversarial robustness on five Atari games compared to state-of-the-art methods.
Simplifies complex pricing models for better interpretability and revenue.
problem Complex pricing models are hard to interpret and not widely adopted.
method Model distillation to create interpretable pricing policies.
result Maximizes revenue while maintaining interpretability.
SDM Policy accelerates inference for robotic tasks while maintaining high action quality.
problem Prolonged inference times in diffusion-based policies for high-frequency control tasks.
method Two-stage optimization: score matching and distribution matching; dual-teacher mechanism.
result 6x inference speedup with state-of-the-art action quality.
CoDistill-GRPO improves small models in GRPO by distilling knowledge from a larger model.
problem Small models in GRPO struggle with sparse rewards on difficult tasks.
method Simultaneously trains a large and small model using co-distillation and GRPO objectives.
result Significant improvement in small model performance over standard GRPO on mathematical benchmarks.
Multiagent reinforcement learning algorithms (MARL) have been demonstrated on complex tasks that require the coordination of a team of multiple agents to complete. Existing works have focused on sharing information between agents via centralized critics to stabilize learning or through communication to increase perform…
A new approach to distill unnormalized EBM for energy-based seq2seq models.
problem Training unnormalized EBM for energy-based seq2seq models is challenging.
method Relating the problem to distributional RL, proposing a general distillation approach.
result General approach applicable to any sequential EBM, illustrated on GAM experiments.
Low precision networks in the reinforcement learning (RL) setting are relatively unexplored because of the limitations of binary activations for function approximation. Here, in the discrete action ATARI domain, we demonstrate, for the first time, that low precision policy distillation from a high precision network pro…
Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tas…
TREK uses distillation to help students solve hard problems.
problem Stalled progress on hard prompts when current policy lacks useful reasoning trajectories.
method TREK combines distillation and reinforcement learning to expand student support.
result TREK significantly improves student performance on mathematical reasoning and agentic tasks.
The paper studies how search and distillation improve reasoning in large language models.
problem Improving reasoning capabilities of large language models.
method Viewing chain-of-thought generation as a metastable Markov process, proving benefits of search and distillation.
result Search protocol rewards sparse edges, reducing the expected number of steps to reach different clusters.
STAR framework reduces OPE variance by distilling complex problems into discrete ARPs.
problem High variance and bias in off-policy evaluation methods.
method STAR framework that includes various OPE estimators and leverages state abstraction.
result Predictions from ARPs estimated from off-policy data are asymptotically correct.
In distributed reinforcement learning, it is common to exchange the experience memory of each agent and thereby collectively train their local models. The experience memory, however, contains all the preceding state observations and their corresponding policies of the host agent, which may violate the privacy of the ag…
Privileged Information Dropout improves RL performance without distillation.
problem Improving sample efficiency and performance in reinforcement learning.
method Introducing Privileged Information Dropout to directly incorporate privileged information into RL agent inputs.
result Privileged Information Dropout outperforms distillation and auxiliary tasks in a partially-observed environment.
New split rules improve subpopulation targeting in policy-making.
problem Improving binary classification for subpopulation targeting in policy-making.
method MDFS, PFS, wEFS for maximizing distance and penalizing final splits.
result Proposed methods target more vulnerable subpopulations than classic CART/KD-CART.
Machine learning has proved to be very successful for making predictions in travel behavior modeling. However, most machine-learning models have complex model structures and offer little or no explanation as to how they arrive at these predictions. Interpretations about travel behavior models are essential for decision…
A method for a single policy to solve various tasks across diverse agent morphologies.
problem Generalizing a single policy to solve various tasks across diverse agent morphologies.
method Unified representation and behavior distillation using a morphology-task graph and Transformer architecture.
result Improves multi-task performances compared to baselines, suggesting a promising approach.
Develops efficient algorithm for distilling supervised and offline RL datasets.
problem Creating synthetic datasets for supervised and offline RL models to match training data performance.
method An efficient algorithm based on matching losses without model training, using randomly sampled regressors.
result Synthetic datasets derived with ildeO(d2) sampled regressors match MSE loss of bounded linear models on training data. Genetic algorithms have been widely used in many practical optimization problems. Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization. However, they have not been shown useful for deep reinforcement learning, possibly …
Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function th…
New findings explain why online methods outperform offline methods in noisy expert feedback settings.
problem The challenge of learning from imperfect expert feedback in sequential decision-making systems.
method Introduced a noisy expert model and a novel variant of on-policy distillation (OPD) to address the gap between offline and online imitation learning.
result Online interaction with a noisy expert via OPD enables polynomial dependence on the horizon, unlike offline methods which require exponential growth in sample complexity.
PPG separates policy and value function training phases for better reinforcement learning efficiency.
problem Challenges in traditional reinforcement learning methods for policy and value function optimization.
method Integrates Phasic Policy Gradient framework that splits policy and value function training into distinct phases.
result Significantly improves sample efficiency on Procgen Benchmark compared to PPO.
Study evaluates off-policy reinforcement learning methods.
problem Evaluating reinforcement learning policies without direct access to the behavior policy.
method Experimental benchmarking suite with diverse design parameters.
result Guidelines for using off-policy evaluation methods in practice.
Curriculum learning and imitation learning improve control over financial time-series data.
problem Improving control performance over complex financial time-series data.
method Data augmentation for curriculum learning and policy distillation for imitation learning.
result Curriculum learning shows significant improvement over time-series control tasks.
Researchers use DT to transfer policies from one environment to another using causal reasoning.
problem Adapting to changes in environmental dynamics in reinforcement learning.
method Applying causal counterfactual reasoning to Decision Transformer (DT) architecture for policy transfer.
result DT successfully transfers a learned policy to new environments while retaining most of the reward.
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert's reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Rece…
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is …
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
LFM learns a sequence of smaller models to generate data from noise.
problem Learning continuous, invertible flows between distributions.
method Stepwise Local Flow Matching (LFM) model, matching diffusion processes up to time-step size.
result LFM achieves competitive generative performance compared to Flow Matching.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robu…