Paper proposes a deep RL method for hedging variable annuities, outperforming misspecified models.
problem Model miscalibration in variable annuity contracts with GMMB and GMDB riders.
method Two-phase deep reinforcement learning approach: training phase in a controlled environment, online learning phase in real market.
result Trained reinforcement learning agent hedges equally well as correct Delta in training phase and outperforms misspecified Deltas.
Deep RL for portfolio management shows poor robustness.
problem Robustness of Deep RL algorithms in online portfolio management.
method Proposed a training and evaluation process for assessing DRL algorithms.
result Most Deep RL algorithms are not robust, generalizing poorly and degrading quickly.
RS-DQN protects RL agents from adversarial attacks.
problem Adversarial attacks can disrupt deep RL training and evaluation.
method Online robustness training with RS-DQN combining state-of-the-art adversarial and provably robust training.
result RS-DQN makes RL agents resilient to strong attacks.
Paper proposes IRL methods for limited interaction scenarios.
problem Learning with limited teacher interaction.
method Curriculum Inverse Reinforcement Learning (CIRL) and Self-Paced Inverse Reinforcement Learning (SPIRL).
result Training strategies can accelerate learning compared to random or batch methods.
Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depe…
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Improved reinforcement learning in Minecraft with human demonstrations.
problem Sample inefficiency in reinforcement learning.
method Training policy networks on human demonstrations first, then fine-tuning with reinforcement learning.
result Best agent achieved a mean score of 48 in Minecraft.
We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation. Our novel technique makes deep reinforcement learning more practical by drastically reducing the trainin…
Hybrid approach combines Markowitz's theory with reinforcement learning for optimal portfolio management.
problem Optimizing investment portfolios while balancing returns and risks.
method Knowledge distillation for training reinforcement learning agents.
result Achieves highest yield and Sharpe ratio of 2.03, ensuring top profitability with low risk.
Reinforcement learning improves game level design.
problem Creating high-quality game levels with limited examples.
method Transforming level design into a Markov decision process and training with reinforcement learning.
result Trained reinforcement learning agents can generate high-quality levels quickly.
IMPACT improves RL training speed without sacrificing sample efficiency.
problem Limited sample efficiency in scalable RL architectures.
method Proposes IMPACT, extending IMPALA with target networks, circular buffers, and truncated importance sampling.
result IMPACT achieves higher rewards and significantly reduces training time compared to IMPALA.
This paper benchmarks batch RL algorithms on Atari, finding DQN and partially-trained policies perform best.
problem Deep RL algorithms fail in batch setting.
method Benchmarked batch RL algorithms on Atari using a single partially-trained policy.
result Many batch RL algorithms underperform DQN and partially-trained policies.
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Deep RL policies are vulnerable to adversarial perturbations, but vanilla training yields more robust policies.
problem Vulnerability of deep reinforcement learning policies to adversarial perturbations.
method Analysis of deep reinforcement learning policy landscape and comparison of vanilla vs. adversarial training.
result Vanilla training yields more robust policies compared to adversarial training.
Generative model solves financial market equilibria with stable reinforcement learning.
problem Financial market equilibria under realistic frictions and multiple agents.
method Generative adversarial reinforcement learning with decoupling feedback.
result Algorithm learns and predicts asset returns and volatilities.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
OpenAI Five defeated Dota 2 champions using deep reinforcement learning.
problem Complex, continuous state-action spaces in Dota 2.
method Deep reinforcement learning, distributed training, self-play.
result Superhuman performance in Dota 2 achieved through reinforcement learning.
Bayesian meta-reinforcement learning improves over point estimates with Laplace approximation.
problem Improving meta-reinforcement learning by providing full posterior distributions.
method Augmenting point estimates with Laplace approximation for full posterior distributions.
result Our method performs similarly to variational baselines with fewer parameters.
Study on meta-reinforcement learning generalization in high-dimensional tasks.
problem Generalization performance of meta-reinforcement learning algorithms in high-dimensional tasks.
method High-dimensional, procedurally generated environments.
result Meta-reinforcement learning algorithms exhibit strong overfitting on challenging tasks.
Mitigates instability in reinforcement learning for safer robotics.
problem Unstable training dynamics in reinforcement learning, especially for safety-sensitive tasks.
method Maintains a history of the agent and reverts to previous parameters when performance decreases.
result Improves performance and stability compared to state-of-the-art algorithms.
New RL method improves robustness across various conditions.
problem Training robust RL agents for diverse environments.
method Adversarial training with Langevin Dynamics.
result Consistently outperforms existing RL algorithms in generalization.
In this paper, we propose a dual memory structure for reinforcement learning algorithms with replay memory. The dual memory consists of a main memory that stores various data and a cache memory that manages the data and trains the reinforcement learning agent efficiently. Experimental results show that the dual memory …
Catalyst.RL accelerates RL research with efficient training.
problem Efficient reinforcement learning training in complex environments.
method Open-source PyTorch framework with distributed training and RL algorithms.
result Catalyst.RL achieved 2nd place in a computationally expensive RL challenge.
Recent work has identified that classification models implemented as neural networks are vulnerable to data-poisoning and Trojan attacks at training time. In this work, we show that these training-time vulnerabilities extend to deep reinforcement learning (DRL) agents and can be exploited by an adversary with access to…
PPOPT uses pretraining to speed up reinforcement learning in physics simulations.
problem High computational costs and inefficiency in reinforcement learning with small training samples.
method A novel policy neural network architecture that combines pretraining and fully-connected networks.
result PPOPT outperforms classic PPO on small training samples in terms of rewards and stability.
Improves reinforcement learning policies for robustness.
problem Lack of robustness in reinforcement learning policies.
method Risk-aware Distributional Reinforcement Learning (SDPG) with CVaR.
result Risk-averse policies achieve robustness against disturbances.
Enhanced reinforcement learning using ensemble methods.
problem Improving reinforcement learning performance.
method Distributional reinforcement learning with ensemble group-aided training.
result Ensemble methods lead to more robust and efficient learning.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
Deep reinforcement learning improves trading performance in financial markets.
problem Improving trading performance in financial markets.
method Deep Q-network (DQN) for designing long-short trading strategies.
result Trained reinforcement learning agent outperformed an index benchmark in trading E-mini S&P 500 futures contracts.
New reinforcement learning bound improves generalization for sequential data.
problem Challenges in obtaining generalization guarantees for reinforcement learning due to sequential data.
method PAC-Bayesian reinforcement learning with consideration of Markov dependencies and mixing time.
result Demonstrated practical utility through PB-SAC, providing meaningful confidence certificates.
New method accelerates Pommerman training with imitation and reinforcement learning.
problem Training complex multi-agent games like Pommerman.
method Imitation learning followed by PPO reinforcement learning with modifications.
result Significantly faster training compared to other methods.
Safe reinforcement learning framework using optimal transport for robustness.
problem Robustness and safety in deep reinforcement learning with limited data assumptions.
method Optimal transport perturbations to construct worst-case virtual state transitions.
result Significantly improved safety at deployment time compared to standard methods.
Study evaluates reinforcement learning algorithms for sequential experimental design.
problem Lack of generalization in reinforcement learning for experimental design.
method Investigated several reinforcement learning algorithms for sequential experimental design.
result Certain algorithms, using dropout or ensemble approaches, show attractive generalization properties.
Surrogate models speed up RL training in dynamic systems.
problem High computational cost of high-fidelity simulations.
method Developed and tested surrogate models for RL training.
result Surrogate models can significantly accelerate RL training.
Paper proposes Cycle-of-Learning framework for better reinforcement learning performance.
problem Efficiently updating policies trained with demonstrations using reinforcement learning.
method Cycle-of-Learning framework combining behavior cloning and 1-step Q-learning losses.
result Cycle-of-Learning framework improves reinforcement learning performance in dense and sparse reward scenarios.
We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the mode…
Study on adversarial training's impact on deep neural reinforcement learning policies.
problem Vulnerability of deep neural reinforcement learning policies to imperceptible adversarial perturbations.
method Two parallel approaches: Fourier spectrum analysis and feature sensitivity measurement.
result Adversarially trained policies are more sensitive to low frequency perturbations.
Framework for multi-agent RL with human feedback in a Snake game.
problem Improving multi-agent reinforcement learning with human feedback.
method Developed a simulated game environment for offline model training and online competitions. Introduced HILL methods and reward manipulation heuristics.
result Agents with HILL methods outperform those without in online competitions.
Deep RL policies share adversarial features across different MDPs.
problem Understanding decision boundaries and loss landscapes in neural policies.
method Investigating similarities in high sensitivity directions across MDPs using Arcade Learning Environment.
result High sensitivity directions for neural policies are correlated across MDPs, suggesting shared non-robust features.
High-throughput 3D control training system achieves 100,000 FPS.
problem Lack of efficient, single-machine reinforcement learning systems.
method Sample Factory combines asynchronous sampling and off-policy correction.
result Achieves 100,000 FPS on 3D control problems without sacrificing sample efficiency.
Explains agent behavior through intended outcomes in reinforcement learning.
problem Proving impossibility of general post-hoc explanations in reinforcement learning.
method Derives local explanations based on intention for Q-function approximations, proving consistency with learned Q-values.
result Demonstrates the necessity of collecting information during training for accurate explanations.
Paper tackles offline meta-reinforcement learning with a new algorithm.
problem Performing reinforcement learning on limited data from a new task.
method Meta-Actor Critic with Advantage Weighting (MACAW) algorithm.
result Achieves notable gains over prior methods on offline meta-RL benchmarks.
Machine learning applications in medical imaging are frequently limited by the lack of quality labeled data. In this paper, we explore the self training method, a form of semi-supervised learning, to address the labeling burden. By integrating reinforcement learning, we were able to expand the application of self train…
Graph Pointer Networks and hierarchical reinforcement learning solve combinatorial optimization problems like TSP.
problem Traveling Salesman Problem (TSP) with constraints.
method Graph Pointer Networks (GPNs) and hierarchical reinforcement learning.
result GPNs and hierarchical RL find optimal solutions for TSP and TSP with time windows.
Procgen Benchmark uses procedurally generated games to test reinforcement learning.
problem Lack of diverse and high-quality training environments for reinforcement learning.
method Developed 16 procedurally generated game-like environments and used them to benchmark reinforcement learning.
result Procedurally generated environments are essential for training and evaluating reinforcement learning agents.
Model-based reinforcement learning (MBRL) has been proposed as a promising alternative solution to tackle the high sampling cost challenge in the canonical reinforcement learning (RL), by leveraging a learned model to generate synthesized data for policy training purpose. The MBRL framework, nevertheless, is inherently…
Framework uses expert intervention to solve long-horizon reinforcement learning tasks.
problem Long horizon robot learning tasks with sparse rewards.
method Option templates and expert intervention to enable high-level task understanding.
result Framework outperforms state-of-the-art approaches by two orders of magnitude.
Training intelligent agents through reinforcement learning is a notoriously unstable procedure. Massive parallelization on GPUs and distributed systems has been exploited to generate a large amount of training experiences and consequently reduce instabilities, but the success of training remains strongly influenced by …