This paper surveys methods to handle non-stationarity in multi-agent deep reinforcement learning.
problem Non-stationarity in multi-agent reinforcement learning environments.
method Modifications in training procedures, opponent policy representation learning, meta-learning, communication, and decentralized learning.
result A comprehensive review of recent works on addressing non-stationarity in multi-agent deep reinforcement learning.
A new framework for multi-agent inverse reinforcement learning.
problem Finding suitable reward functions for multi-agent scenarios.
method MA-AIRL, based on a new solution concept and maximum pseudolikelihood estimation.
result MA-AIRL can recover highly correlated reward functions and significantly outperforms prior methods.
In this paper, we explore using deep reinforcement learning for problems with multiple agents. Most existing methods for deep multi-agent reinforcement learning consider only a small number of agents. When the number of agents increases, the dimensionality of the input and control spaces increase as well, and these met…
This paper uses deep reinforcement learning to optimize stock liquidation strategies.
problem Optimizing liquidation strategies in the stock market considering market impact and trader risk.
method Multi-agent deep reinforcement learning model to learn optimal selling decisions.
result Reinforcement learning models can effectively solve realistic liquidation problems.
New algorithm for multi-agent reinforcement learning with reduced communication.
problem Cooperative learning among multiple agents with limited communication.
method Randomized multi-agent actor-critic algorithm for directed graphs.
result Algorithm solves problem for strongly connected graphs with reduced communication.
Improves efficiency and scalability in multi-agent reinforcement learning.
problem Non-stationarity and inefficiency in critic networks due to agent permutations.
method Proposes a permutation invariant critic (PIC) to avoid changes in critic output due to agent permutations.
result Achieves improvements of test episode reward between 15% to 50% on challenging multi-agent particle environment (MPE).
PettingZoo library accelerates multi-agent reinforcement learning research.
problem Challenges in multi-agent reinforcement learning, especially conceptual models of games.
method Developed PettingZoo library with AEC games model to address multi-agent reinforcement learning challenges.
result AEC games model addresses conceptual issues in multi-agent reinforcement learning environments.
Paper develops a new multi-agent reinforcement learning algorithm.
problem Improving policies in a network of communicating agents.
method Develops a multi-agent off-policy actor-critic algorithm using emphatic temporal difference learning.
result Proves convergence of the algorithm under linear function approximation.
A novel framework uses goal-conditioned reinforcement learning to generate diverse samples.
problem Generating high-quality, diverse samples from generative models.
method Two agents: GC-agent learns to reconstruct the training set, S-agent learns to imitate GC-agent without knowing the goals.
result Empirically, the method generates diverse and high-quality samples in image synthesis.
Deep RL agents vary significantly in Atari environments.
problem Challenges in reproducibility due to stochasticity.
method Experiments with OpenAI Baselines agents.
result Variability in agent performance is significant and underreported.
Study models crypto markets using multi-agent reinforcement learning.
problem Emulating crypto market dynamics and behaviors.
method Multi-agent reinforcement learning (MARL) with RL techniques.
result Model accurately emulates crypto market microstructure and behaviors.
Mobile agents classify images via reinforcement learning and consensus.
problem Image classification using multiple mobile agents.
method Proposed network architecture for local belief formation and feature extraction. Decentralized consensus protocol using reinforcement learning.
result Effectiveness of the proposed framework demonstrated on MNIST dataset.
A lifelong learning architecture improves reinforcement learning policies using simulations and a DNC model.
problem Improving reinforcement learning policies in dynamic environments.
method Iterative training of a Reinforcement Learning agent and a DNC model in conjunction.
result DNC models can continually learn from pixels alone to simulate new tasks.
Attackers can significantly reduce team rewards in cooperative multi-agent reinforcement learning.
problem Robustness of cooperative multi-agent reinforcement learning to adversaries.
method Novel attack method involving training a policy network and using targeted adversarial examples.
result Reduces team reward from 20 to 9.4 by attacking a single agent, reducing winning rate from 98.9% to 0%.
New snooping attacks exploit deep RL without access to environment.
problem Security vulnerabilities in deep reinforcement learning.
method Proposes snooping threat models and attacks on RL agents.
result Adversaries can launch attacks without interacting with the environment.
Deep learning has enabled traditional reinforcement learning methods to deal with high-dimensional problems. However, one of the disadvantages of deep reinforcement learning methods is the limited exploration capacity of learning agents. In this paper, we introduce an approach that integrates human strategies to increa…
Paper tackles delays in multi-agent reinforcement learning, improving performance.
problem Challenges in reinforcement learning due to delays in real-world systems.
method Proposes a novel framework for multi-agent reinforcement learning with delays, using Delay-Aware Markov Games and centralized-decentralized training.
result Demonstrates significant improvement in performance with delay-aware multi-agent reinforcement learning.
CODA resolves coordination issues in offline multi-agent reinforcement learning.
problem Coordination failure in offline multi-agent reinforcement learning.
method Diffusion-based multi-agent trajectory generator for data augmentation.
result CODA resolves coordination pathologies in continuous polynomial games and complex benchmarks.
This paper uses counterfactual thinking to improve multi-agent reinforcement learning.
problem Improving decision-making in multi-agent environments.
method Proposes a deep reinforcement learning model with counterfactual thinking to generate and evaluate multiple actions.
result Counterfactual thinking enhances agents' performance in multi-agent environments, leading to higher rewards and fair competition.
Two policy regularization methods improve coordination in multi-agent reinforcement learning.
problem Challenging to discover successful collective behaviors in multi-agent reinforcement learning.
method Proposes TeamReg and CoachReg methods to promote coordination.
result Improved performance across cooperative multi-agent problems.
Framework for multi-agent RL with human feedback in a Snake game.
problem Improving multi-agent reinforcement learning with human feedback.
method Developed a simulated game environment for offline model training and online competitions. Introduced HILL methods and reward manipulation heuristics.
result Agents with HILL methods outperform those without in online competitions.
New algorithm reduces complexity in multi-agent reinforcement learning.
problem High computational complexity in exact computations for multi-agent reinforcement learning.
method Design of a scalable algorithm based on Natural Policy Gradient, using local information and limited communication.
result Converges to globally optimal policy with dimension-free complexity and localization error.
New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
Reinforcement learning crypto agent achieves high returns on Bitcoin derivatives.
problem Maximizing returns on volatile cryptocurrency markets.
method Online transfer learning with an echo state network and recurrent reinforcement learning.
result Achieves a total return of 350%, net of transaction costs, over five years.
MARLA uses deep reinforcement learning for multi-agent AHT, reducing Bayes risk.
problem Decentralized active hypothesis testing among multiple agents.
method MARLA algorithm using deep multi-agent reinforcement learning.
result MARLA outperforms single-agent learning in AHT problems.
Paper tackles RL for power grid topology optimization.
problem Managing large action spaces in growing power networks.
method Hierarchical multi-agent reinforcement learning (MARL) framework.
result MARL framework outperforms single-agent RL methods.
In this paper, reinforcement learning is applied to the problem of optimizing market making. A multi-agent reinforcement learning framework is used to optimally place limit orders that lead to successful trades. The framework consists of two agents. The macro-agent optimizes on making the decision to buy, sell, or hold…
Deep RL fails on deceptive games, revealing algorithm weaknesses.
problem Characterizing and understanding failures of deep reinforcement learning.
method Testing A2C on four deceptive games using a game framework.
result Deep RL fails in specific ways that differ from planning-based agents.
MARL algorithm uses regularization to avoid explicit structures, improving performance.
problem Lack of effective reinforcement learning methods for multi-agent systems.
method MARQ uses regularization to promote structured exploration without explicit centralized structures.
result MARQ outperforms existing methods in multi-agent environments.
REFIL learns from imagined sub-group interactions to improve multi-agent reinforcement learning.
problem Learning efficiency in multi-agent reinforcement learning settings.
method Randomized Entity-wise Factorization (REFIL) approach.
result REFIL outperforms all strong baselines in challenging multi-task StarCraft settings.
This work formalizes and extends parameter sharing in multi-agent reinforcement learning.
problem Parameter sharing limits multi-agent learning to a single policy, preventing different tasks or action spaces.
method Introduces agent indication and extends parameter sharing to heterogeneous observation and action spaces.
result Proves convergence to optimal policies for parameter sharing in heterogeneous environments.
A framework helps reinforcement learning agents understand and decompose tasks from human demonstrations.
problem Sparse-reward tasks where demonstrations are used as sources of causal knowledge.
method Develops causal models through observation and reasons from this knowledge to decompose tasks.
result A basic implementation of Reasoning from Demonstration (RfD) is effective in sparse-reward tasks.
MaxMax Q-Learning improves coordination in multi-agent reinforcement learning by refining action selection.
problem Relative over-generalization in decentralized multi-agent reinforcement learning.
method MaxMax Q-Learning employs iterative sampling and evaluation of potential next states to refine approximations of ideal state transitions.
result MaxMax Q-Learning frequently outperforms existing baselines, demonstrating enhanced convergence and sample efficiency.
Adversaries can manipulate cooperative MARL networks.
problem Fragility of consensus-based MARL algorithms to adversarial attacks.
method Study of adversarial attacks on consensus-based MARL networks.
result Consensus-based MARL algorithms are vulnerable to adversarial manipulation.
Distributed learning method for multi-agent reinforcement learning with policy coordination.
problem Solving multi-agent reinforcement learning problems with coordination.
method Distributed off-policy actor critic with policy consensus.
result The proposed algorithm achieves asymptotic agreement on the global optimal policy function.
We present an effective technique for training deep learning agents capable of negotiating on a set of clauses in a contract agreement using a simple communication protocol. We use Multi Agent Reinforcement Learning to train both agents simultaneously as they negotiate with each other in the training environment. We al…
Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn. Given the importance of learning in our intelligence, RL has been thought to be one of key compone…
Method models other agents' behaviors without requiring direct observation.
problem Understanding and interacting effectively with other agents in reinforcement learning.
method Extracts representations from local observations of the controlled agent using encoder-decoder architectures.
result The method achieves higher returns than baseline methods in multi-agent environments.
Deep Reinforcement Learning has been shown to be very successful in complex games, e.g. Atari or Go. These games have clearly defined rules, and hence allow simulation. In many practical applications, however, interactions with the environment are costly and a good simulator of the environment is not available. Further…
This review covers recent cooperative MARL approaches and their challenges.
problem Recent progress in multi-agent reinforcement learning (MARL).
method Five common approaches: independent learners, fully observable critic, value function factorization, consensus, and learn to communicate.
result Challenges and solutions for each approach are discussed.
Improved PAC guarantees for multi-agent reinforcement learning with noisy communication.
problem Improving exploration in cooperative multi-agent reinforcement learning with communication constraints.
method Develops PAC guarantees for multiple concurrent MDPs with noisy and resource-limited communication.
result Theoretical and empirical improvements in sample complexity for information fusion.
A new method simulates large, diverse populations of learning agents evolving in games.
problem Limited scalability and efficiency of Multi-Agent Reinforcement Learning.
method Parallelizable implementation of Policy Gradient and Opponent-Learning Awareness for evolutionary simulations.
result Simulated large, diverse populations of learning agents evolve under various strategies.
Proposes using logical specifications for multi-objective reinforcement learning to improve agent behavior.
problem Difficulties in controlling reinforcement learning agents and the need for better generalization.
method Uses propositional logic to specify the importance of multiple objectives, encoding these specifications using a recurrent neural network.
result MORL agents parameterized by logical specifications can generalize to novel combinations of objectives and achieve comparable performance.
RL agent learns to save costs by managing household energy storage.
problem Maximizing cost savings in smart grids with household energy storage.
method Data-driven RL agent learns from tariff structures and storage capacity.
result RL agent explains its learning process and strategies.
Improved exploration in cooperative multi-agent reinforcement learning.
problem Limited expressiveness of Gaussian policies in DecSPG hinders effective exploration.
method Proposes decentralized diffusion policy learning (DDPL) with denoising diffusion probabilistic models.
result Consistently improved performance on various MARL benchmarks.
Due to the capability of deep learning to perform well in high dimensional problems, deep reinforcement learning agents perform well in challenging tasks such as Atari 2600 games. However, clearly explaining why a certain action is taken by the agent can be as important as the decision itself. Deep reinforcement learni…
This paper proposes a decentralized reinforcement learning method for multi-agent resource allocation.
problem Allocating heterogeneous resources among multiple agents in a decentralized manner.
method Liquid-Graph-Time Clustering-IPPO, integrating dynamic cluster consensus.
result LGTC-IPPO achieves more stable rewards, better coordination, and robust performance.
RL agent learns to manage inventory and price in dealer market simulations.
problem Managing inventory and price in a dealer market with RL.
method Multi-agent simulation, reinforcement learning, different reward formulations.
result RL agent learns competitor's pricing and manages inventory effectively.