TOYBOX creates new Atari environments for deep RL experiments.
problem Challenges in evaluating deep RL agent behavior.
method Designs and opensource releases a subset of Atari environments.
result TOYBOX enables experiments impossible in other environments.
Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learn…
RL agent learns to smoothly change lanes in a dynamic driving environment.
problem Challenging lane change control with safety and comfort.
method Formulated continuous action for lane change in DDPG algorithm, defined reward function for learning.
result Successfully changed lanes with 100% success rate in diverse driving situations.
New RL framework simulates financial market dynamics.
problem Complex financial market dynamics under various scenarios.
method Two RL families learn simultaneously, using Deep RL and parametrized reward.
result Agents learn a shared policy for diverse behaviors.
Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architec…
Study uses RL to simulate realistic market behavior.
problem Traditional market simulators lack realistic dynamic behavior.
method Agent-based simulation with reinforcement learning agents.
result RL agents simulate realistic stylized facts and market behavior.
ABPS improves RL training efficiency by sharing policies and evolving hyper-params.
problem Data inefficiency in training deep RL models for real-world applications.
method ABPS: adaptive behavior policy sharing; ABPS-PBT: hybridizing ABPS with PBT for evolving hyper-params.
result ABPS achieves superior performance and reduced variance compared to conventional hyper-parameter tuning.
Adversarial policies can defeat RL agents in multi-agent environments.
problem Vulnerability of RL policies to adversarial perturbations.
method Demonstrated adversarial policies in a multi-agent, high-dimensional environment.
result Adversarial policies reliably win against RL agents but generate seemingly random behavior.
RL agents learn to detect and mitigate adversarial attacks.
problem Adversarial attacks on Deep RL algorithms deployed in real-life applications.
method Meta-Learned Advantage Hierarchy (MLAH) agent using meta-learning for online robustness.
result The MLAH agent exhibits hierarchical coping behaviors and maintains higher reward distributions over time.
We build deep RL agents that execute declarative programs expressed in formal language. The agents learn to ground the terms in this language in their environment, and can generalize their behavior at test time to execute new programs that refer to objects that were not referenced during training. The agents develop di…
Deep RL for multi-agent autonomous driving in dynamic environments.
problem Adapting to dynamic, multi-agent driving environments.
method Formulated Partially Observable Markov Games (POSG) for multi-agent learning.
result Demonstrated successful training of multi-agent control policies.
This study benchmarks AI agents for personalized retail promotions using simulations.
problem Optimizing coupon targeting for sparse customer purchase events.
method Comprehensive simulations of customer shopping behaviors; training RL agents on batch data.
result Contextual bandit and deep RL methods outperform static policies in sparse reward environments.
Study shows how to motivate AI agents like humans through learning from demonstrations.
problem Transfer human-like intrinsic motivation to AI agents in RL.
method Inverse RL approach to learn an exploration bonus from demonstrations.
result AI agents can learn and use complex exploration behaviors from demonstrations.
RADIAL-RL improves deep RL agents' robustness against adversarial attacks.
problem Vulnerability of deep reinforcement learning agents to small adversarial perturbations.
method RADIAL-RL, a principled framework for training robust reinforcement learning agents.
result RADIAL-RL-trained agents consistently outperform prior methods in robustness tests.
We tackle the blackbox issue of deep neural networks in the settings of reinforcement learning (RL) where neural agents learn towards maximizing reward gains in an uncontrollable way. Such learning approach is risky when the interacting environment includes an expanse of state space because it is then almost impossible…
RL agent learns to avoid market spoofing.
problem Avoiding subtle non-normative behavior in trading agents.
method Learned recognizer incorporated into RL agent's reward function.
result RL agent avoids spoofing while remaining profitable.
RL agent learns to save costs by managing household energy storage.
problem Maximizing cost savings in smart grids with household energy storage.
method Data-driven RL agent learns from tariff structures and storage capacity.
result RL agent explains its learning process and strategies.
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, an…
Deep RL fails on deceptive games, revealing algorithm weaknesses.
problem Characterizing and understanding failures of deep reinforcement learning.
method Testing A2C on four deceptive games using a game framework.
result Deep RL fails in specific ways that differ from planning-based agents.
RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
FEPS models agents to learn and act in complex environments without deep neural networks.
problem Modeling complex adaptive systems and understanding self-organizing behavior.
method Introducing Free Energy Projective Simulation (FEPS) within the constraints of the free energy principle and active inference.
result FEPS agents resolve ambiguity and infer optimal policies in partially observable environments.
Combining global and local explanations improves user understanding of RL agents.
problem Challenges in explaining agent behavior due to large state spaces and delayed rewards.
method Integrating strategy summaries with saliency maps to provide both global and local explanations.
result Summaries including important states significantly improve user understanding of RL agents.
We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions t…
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use param…
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
Study multi-agent RL in OTC markets, learning from agents' interactions.
problem Designing efficient RL solutions for OTC market interactions.
method Parameterized reward functions, shared policy learning, RL calibration.
result Agents learn to balance hedging and skewing in market simulations.
Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn. Given the importance of learning in our intelligence, RL has been thought to be one of key compone…
Agents learn social skills from each other, improving performance.
problem Independent RL agents fail to use social learning.
method Imposed constraints, model-based auxiliary loss, mixed training.
result Agents trained with social learning outperform solo-trained agents.
Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision-making problems. These algorithms however have faced great challenges when dealing with high-dimensional environments. The recent development of deep learning has enabled RL methods to drive optimal pol…
Adaptive policies for multi-agent RL improve adaptiveness in changing environments.
problem Non-stationarity in multi-agent reinforcement learning where other agents may alter their policies.
method Train multiple adaptive policies for each agent and a policy predictor to select the best policy at execution time.
result Agents trained with our method outperform state-of-the-art methods in all tested environments.
In this work we describe a novel deep reinforcement learning architecture that allows multiple actions to be selected at every time-step in an efficient manner. Multi-action policies allow complex behaviours to be learnt that would otherwise be hard to achieve when using single action selection techniques. We use both …
Deep RL improves Diplomacy performance, outperforming previous methods.
problem Applying RL to complex, many-agent, simultaneous-move games like Diplomacy.
method Best response policy iteration and fictitious play approximation.
result Deep RL agents convincingly outperform previous Diplomacy agents.
This paper presents a novel approach to the technical analysis of wireheading in intelligent agents. Inspired by the natural analogues of wireheading and their prevalent manifestations, we propose the modeling of such phenomenon in Reinforcement Learning (RL) agents as psychological disorders. In a preliminary step tow…
Framework analyzes RL agents' behavior to explain their strengths and weaknesses.
problem Understanding and explaining RL agents' capabilities and limitations.
method Data analysis and visualization of interaction history to extract interestingness elements.
result Visual summaries help humans correctly perceive agents' strengths and weaknesses.
Interprets how intrinsic motivation shapes behavior in RL agents.
problem Understanding how intrinsic motivation influences behavior in reinforcement learning agents.
method Analyzed five RL agents in procedurally generated environments using various interpretability techniques.
result Curiosity-driven agents exhibit broader and more dynamic attention than extrinsically motivated agents.
Deep RL agent improves lane changing in unpredictable traffic.
problem Uncertainty in other drivers' behaviors and safety vs agility trade-off.
method Developed a deep reinforcement learning agent in a simulated highway environment.
result Significantly better performance in noisy environments compared to heuristic methods.
Improves deep RL agents' generalization to unseen environments.
problem Deep RL agents struggle to adapt to new environments.
method Information Bottleneck regularization and annealing-based optimization.
result Agents can generalize to test parameters more than 10 standard deviations away from training.
New RL approach tackles constrained Markov decision processes.
problem Applying RL to physical systems with safety constraints.
method Formulated as a Constrained Markov Decision Process (CMDP), introduced a safe policy improvement method.
result Agent learns to maximize returns while satisfying constraints.
Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action sp…
Randomized neural networks improve deep RL agents' generalization.
problem Deep RL agents struggle to generalize to new, semantically similar environments.
method Introduce a randomized (convolutional) neural network to perturb input observations.
result Significantly outperforms various regularization and data augmentation methods.
We build a deep reinforcement learning (RL) agent that can predict the likelihood of an individual testing positive for malaria by asking questions about their household. The RL agent learns to determine which survey question to ask next and when to stop to make a prediction about their likelihood of malaria based on t…
MASA framework uses RL to balance portfolio returns and risks.
problem Managing portfolio risk in turbulent financial markets.
method Multi-agent reinforcement learning with a market observer.
result MASA framework outperforms RL approaches in balancing returns and risks.
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effective…
Automatically generates a deep RL curriculum for faster and more stable learning.
problem How to automatically generate a curriculum for deep RL agents.
method Interprets curriculum generation as an inference problem, learning task distributions progressively.
result Curricula significantly improve learning performance across various environments and deep RL algorithms.
Enhances RL agents with predictive internal representations.
problem Improving model-free reinforcement learning agents.
method Introduces Deep InfoMax (DIM) objective to train predictive internal representations.
result Successfully learned predictive representations in synthetic settings.
Gray-box attack improves on white-box methods for trading agents.
problem Robustness of Deep RL trading agents against adversarial attacks.
method Hybrid Deep Neural Network policy for gray-box adversarial attack.
result Adversary can reduce trading agent's reward by 214.17%.
RH-UCRL combines pessimism and optimism for robust RL.
problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.
method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.
result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.
Deep RL agents suffer from transient non-stationarity, which ITER mitigates.
problem Transient non-stationarity in deep RL agents affects generalization.
method Iterated Relearning (ITER) transfers knowledge between networks to reduce non-stationarity.
result ITER improves deep RL agents' performance on generalization benchmarks.