RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn. Given the importance of learning in our intelligence, RL has been thought to be one of key compone…
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
Motivated by recent advancements in Deep Reinforcement Learning (RL), we have developed an RL agent to manage the operation of storage devices in a household and is designed to maximize demand-side cost savings. The proposed technique is data-driven, and the RL agent learns from scratch how to efficiently use the energ…
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computa…
Reincarnating RL reuses prior work to accelerate RL progress.
problem Efficiency and accessibility in reinforcement learning for large-scale applications.
method Transfer of learned policies between RL agents or design iterations, focusing on value-based RL.
result Demonstrated gains in performance over tabula rasa RL on various tasks.
Recent research on reinforcement learning (RL) has suggested that trained agents are vulnerable to maliciously crafted adversarial samples. In this work, we show how such samples can be generalised from White-box and Grey-box attacks to a strong Black-box case, where the attacker has no knowledge of the agents, their t…
RADIAL-RL improves deep RL agents' robustness against adversarial attacks.
problem Vulnerability of deep reinforcement learning agents to small adversarial perturbations.
method RADIAL-RL, a principled framework for training robust reinforcement learning agents.
result RADIAL-RL-trained agents consistently outperform prior methods in robustness tests.
RL agents optimize order execution in a realistic market simulation.
problem Optimal order execution challenges in a complex market.
method Multi-agent RL in a historical order book simulation.
result RL agents converge to TWAP strategies in some scenarios.
The paper uses RL to verify hypotheses, overcoming existing limitations.
problem Verifying hypotheses using reinforcement learning.
method Formulated hypothesis verification as an RL problem, exploiting hypothesis structure.
result RL agents can successfully verify hypotheses, even those not factorizable.
Paper tackles RL for power grid topology optimization.
problem Managing large action spaces in growing power networks.
method Hierarchical multi-agent reinforcement learning (MARL) framework.
result MARL framework outperforms single-agent RL methods.
Study uses RL to simulate realistic market behavior.
problem Traditional market simulators lack realistic dynamic behavior.
method Agent-based simulation with reinforcement learning agents.
result RL agents simulate realistic stylized facts and market behavior.
Survey of RL algorithms for dynamic environments.
problem Handling changing operating conditions in RL.
method Survey of algorithms developed for non-stationary environments.
result Methods to adapt to varying operating conditions.
Study differential privacy in multi-agent RL, achieving efficient and private learning.
problem Protecting sensitive data in multi-agent reinforcement learning.
method Extending DP definitions to two-player games, designing an efficient algorithm with privatized bonuses.
result Achieved trajectory-wise differential privacy in multi-agent RL, improving regret bounds.
We build a deep reinforcement learning (RL) agent that can predict the likelihood of an individual testing positive for malaria by asking questions about their household. The RL agent learns to determine which survey question to ask next and when to stop to make a prediction about their likelihood of malaria based on t…
Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architec…
RH-UCRL combines pessimism and optimism for robust RL.
problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.
method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.
result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.
RL framework optimizes trading costs in noisy markets.
problem Optimal execution and placement in noisy markets.
method Dual-window Denoise PPO RL network, imitation learning, comprehensive market features, flexible action formulation.
result RL agents outperformed TWAP strategy in execution cost.
We propose a novel approach to address one aspect of the non-stationarity problem in multi-agent reinforcement learning (RL), where the other agents may alter their policies due to environment changes during execution. This violates the Markov assumption that governs most single-agent RL methods and is one of the key c…
Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action sp…
Deep RL improves Diplomacy performance, outperforming previous methods.
problem Applying RL to complex, many-agent, simultaneous-move games like Diplomacy.
method Best response policy iteration and fictitious play approximation.
result Deep RL agents convincingly outperform previous Diplomacy agents.
We aim to jointly optimize antenna tilt angle, and vertical and horizontal half-power beamwidths of the macrocells in a heterogeneous cellular network (HetNet). The interactions between the cells, most notably due to their coupled interference render this optimization prohibitively complex. Utilizing a single agent rei…
PI-SAC agents learn predictive information to improve RL efficiency.
problem Improving sample efficiency in reinforcement learning.
method PI-SAC agents use a contrastive version of Conditional Entropy Bottleneck to learn predictive information from past and future states.
result PI-SAC agents significantly improve sample efficiency on challenging continuous control tasks.
Agents learn social skills from each other, improving performance.
problem Independent RL agents fail to use social learning.
method Imposed constraints, model-based auxiliary loss, mixed training.
result Agents trained with social learning outperform solo-trained agents.
New RL algorithms reduce costs for single-agent and federated learning.
problem Minimizing costs in RL and federated RL settings.
method Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost algorithms.
result First algorithms to achieve low burn-in and logarithmic switching costs.
Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision-making problems. These algorithms however have faced great challenges when dealing with high-dimensional environments. The recent development of deep learning has enabled RL methods to drive optimal pol…
MASA framework uses RL to balance portfolio returns and risks.
problem Managing portfolio risk in turbulent financial markets.
method Multi-agent reinforcement learning with a market observer.
result MASA framework outperforms RL approaches in balancing returns and risks.
New algorithm finds Nash equilibrium in multi-agent games.
problem Extending RL theory to multi-agent settings with large state spaces.
method Proposes an algorithm using an exploiter to find Nash equilibrium policies.
result Proves finding Nash equilibrium is possible with polynomial samples.
Robust RL with learned optimal adversary improves agent performance under adversarial state observations.
problem Ensuring reinforcement learning agents' robustness against adversarial perturbations of state observations.
method Proposed a framework of alternating training with learned adversaries (ATLA) to find optimal adversarial policies and enhance agent robustness.
result ATLA achieves state-of-the-art performance under strong adversaries in continuous control environments.
CARL safely adapts RL agents for safety-critical tasks.
problem Safety hazards in RL for safety-critical tasks.
method CARL combines model-based RL and cautious adaptation.
result CARL achieves higher rewards with fewer failures in safety-critical tasks.
Improves deep RL agents' generalization to unseen environments.
problem Deep RL agents struggle to adapt to new environments.
method Information Bottleneck regularization and annealing-based optimization.
result Agents can generalize to test parameters more than 10 standard deviations away from training.
RL improves market making with historical data time travel.
problem Limited ability to simulate and fully appraise the impact of actions in competitive systems.
method Introduces 'consistent data time travel' to adjust historical data time index.
result Significant improvement in agent's gain with data time travel.
Agents learn to give rewards to others in a shared learning environment.
problem How to encourage cooperation among RL agents in a shared environment.
method Each agent learns a reward function to influence others, optimizing for its own and others' extrinsic objectives.
result Agents significantly outperform standard RL in Markov games, often finding near-optimal division of labor.
New method provides scalable safety guarantees for RL agents.
problem Safe reinforcement learning in real-life scenarios.
method State-augmentation and shield design for probabilistic avoidance.
result Strict formal safety guarantees for RL agents, scalable and practical.
Paper introduces a novel reward function for noisy financial markets using imitation learning.
problem Noisy reward function in financial markets hinders RL agent performance.
method Integrates imitation learning feedback with reinforcement learning to improve reward function design.
result Improves financial performance metrics compared to traditional benchmarks and RL agents.
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
This study proposes hidden state curiosity to enhance RL models' resilience against noise.
problem Curiosity traps in RL models distract agents from discovering novel experiences.
method Proposed hidden state curiosity based on the Free Energy Principle to reward agents for KL divergence between predictive priors and posteriors.
result Agents with hidden state curiosity are more resilient against curiosity traps compared to those with prediction error curiosity.
Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learn…
AI traders learn to exploit meta-orders from slower traders, increasing their profits.
problem Adverse selection of medium-frequency traders by high-frequency AI agents.
method Reinforcement learning in a Hawkes LOB model, with impulse control and PPO.
result AI agents can learn to capitalize on meta-orders, increasing their profits.
Cross-dimensional neural networks improve AI in Catan game.
problem Challenging to build AI agents for Catan game using RL.
method Introduced cross-dimensional neural networks to handle game complexities.
result RL agent outperforms best heuristic agent in Catan.
RL in MFGs is as hard as solving many single-agent RL problems.
problem Learning Nash Equilibrium in Mean-Field Games (MFGs).
method Introduce P-MBED to measure model complexity, develop a novel exploration strategy, and establish polynomial sample complexity results.
result Learning Nash Equilibrium in MFGs is no more statistically challenging than solving a logarithmic number of single-agent RL problems.
RL agent learns to avoid market spoofing.
problem Avoiding subtle non-normative behavior in trading agents.
method Learned recognizer incorporated into RL agent's reward function.
result RL agent avoids spoofing while remaining profitable.
Study investigates key design choices in on-policy RL algorithms.
problem Lack of transparency in RL algorithm implementations.
method Implemented >50 design choices in a unified RL framework.
result Insights and recommendations for on-policy RL training.
GraSP-RL uses graph neural networks to improve job shop scheduling.
problem Capturing machine-unit-job sequence relationships and managing state space growth.
method Graph neural networks for feature extraction, reinforcement learning for decision-making, decentralized optimization.
result GraSP-RL outperforms existing methods in minimizing makespan for complex production environments.
A new buffer system improves continual learning in RL agents by adapting to changing environments.
problem Improving RL agents' ability to learn from changing environments over time.
method Multi-timescale replay buffer combined with invariant risk minimization.
result The method shows improvement over baselines in continual learning settings.
In the last few years, deep multi-agent reinforcement learning (RL) has become a highly active area of research. A particularly challenging class of problems in this area is partially observable, cooperative, multi-agent learning, in which teams of agents must learn to coordinate their behaviour while conditioning only…
Market makers play an important role in providing liquidity to markets by continuously quoting prices at which they are willing to buy and sell, and managing inventory risk. In this paper, we build a multi-agent simulation of a dealer market and demonstrate that it can be used to understand the behavior of a reinforcem…
Language helps RL agents learn complex relational and causal structures.
problem Learning relational and causal structure in complex environments.
method Training RL agents to predict language descriptions and explanations.
result Language aids agents in learning challenging relational and causal tasks.