MARLeME extracts MARL models into symbolic models for better interpretability.
problem Improving interpretability and explainability of MARL systems.
method Develops a MARL model extraction library that approximates MARL systems with symbolic models.
result Enhanced understanding and inspection of MARL systems and agents.
New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
GUM tackles MARL by avoiding overestimation through state-marginal restriction.
problem Overestimation of values in large joint state-action spaces.
method Greedy UnMixing through state-marginal restriction and unmixing.
result Superior performance compared to existing Q-learning and general MARL algorithms.
Paper develops MFC approach to reduce MARL complexity.
problem Curse of dimensionality in cooperative MARL.
method Mean-field control (MFC) and Q-learning.
result Linear convergence rate for MFC-K-Q, independent of N N N . Despite the increasing interest in multi-agent reinforcement learning (MARL) in multiple communities, understanding its theoretical foundation has long been recognized as a challenging problem. In this work, we address this problem by providing a finite-sample analysis for decentralized batch MARL with networked agents…
Paper proves efficiency of MARL with transformers, addressing agent complexity.
problem Theoretical understanding of MARL with many agents and limited relational reasoning.
method Set Transformer for relational reasoning, model-free and model-based MARL algorithms.
result Provable efficiency of MARL algorithms, suboptimality gaps independent of number of agents.
The paper reviews MARL and its causal challenges, advocating for a 'causality first' approach.
problem Challenges in multi-agent reinforcement learning (MARL) and lack of theoretical guarantees for emergent behavior.
method Discussion of how causal methods can improve safety, interpretability, and robustness in MARL.
result Causal methods can provide strong theoretical guarantees for emergent behavior in MARL.
Adversaries can manipulate cooperative MARL networks.
problem Fragility of consensus-based MARL algorithms to adversarial attacks.
method Study of adversarial attacks on consensus-based MARL networks.
result Consensus-based MARL algorithms are vulnerable to adversarial manipulation.
ABIDES-MARL uses MARL to study market behavior in a realistic financial simulation.
problem Understanding equilibrium behavior in complex financial market games.
method Combines MARL with a realistic LOB simulation to study market behavior.
result Validated approach by solving an extended Kyle model and showing how execution strategies shape market dynamics.
This chapter reviews recent advances in multi-agent reinforcement learning.
problem Theoretical foundations for multi-agent reinforcement learning are lacking.
method Selective overview of MARL algorithms with theoretical analysis.
result Identification of new research directions in MARL theory.
Introduces LTQL for factored policies in cooperative MARL.
problem Learning optimal joint policies in collaborative MARL scenarios.
method Logical Team Q-learning (LTQL) as a stochastic approximation to dynamic programming.
result LTQL provides factored policies for optimal joint behavior in cooperative MARL.
Study compares MAPF and MARL algorithms for warehouse automation.
problem Optimizing multi-agent pickup and delivery in warehouse settings.
method Compared conflict-based search (MAPF) and shared experience actor-critic (MARL).
result Comprehensive benchmarking of MAPF and MARL in a simulated warehouse environment.
New methods improve multi-agent reinforcement learning by addressing rotational dynamics.
problem Reproducibility crisis in multi-agent reinforcement learning.
method Reframing MARL approaches using Variational Inequalities (VIs) and proposing gradient-based VI methods.
result Significant performance improvements across benchmarks, including better convergence to equilibrium strategies in zero-sum games.
AEC Games model represents software MARL environments better than POSGs.
problem POSGs are conceptually unsuitable for software MARL environments.
method Introduced AEC Games model as an equivalent to POSGs.
result AEC Games model is more representative of software MARL environments.
Multi-agent reinforcement learning (MARL) has recently received considerable attention due to its applicability to a wide range of real-world applications. However, achieving efficient communication among agents has always been an overarching problem in MARL. In this work, we propose Variance Based Control (VBC), a sim…
This paper improves MARL for networked systems through new protocols and discount factors.
problem Improving control in networked systems using multi-agent reinforcement learning.
method Formulated as a spatiotemporal Markov decision process, introduced a spatial discount factor, and proposed NeurComm.
result Appropriate spatial discount factor enhances learning curves of non-communicative MARL algorithms.
Exploration efficiency is a challenging problem in multi-agent reinforcement learning (MARL), as the policy learned by confederate MARL depends on the collaborative approach among multiple agents. Another important problem is the less informative reward restricts the learning speed of MARL compared with the informative…
PettingZoo library accelerates multi-agent reinforcement learning research.
problem Challenges in multi-agent reinforcement learning, especially conceptual models of games.
method Developed PettingZoo library with AEC games model to address multi-agent reinforcement learning challenges.
result AEC games model addresses conceptual issues in multi-agent reinforcement learning environments.
MARL improves LBM stability and accuracy across scales.
problem Stability and accuracy issues in under-resolved LBM simulations.
method Multi-Agent Reinforcement Learning (MARL) to dynamically control local relaxation parameters.
result MARL closures stabilize simulations and recover spectra of fully resolved models.
Deep Reinforcement Learning has made significant progress in multi-agent systems in recent years. In this review article, we have focused on presenting recent approaches on Multi-Agent Reinforcement Learning (MARL) algorithms. In particular, we have focused on five common approaches on modeling and solving cooperative …
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones f…
Flexible decentralized MARL framework for cooperative multi-agent learning.
problem Complexity and impracticality of centralized MARL in complicated applications.
method Flexible fully-decentralized actor-critic MARL framework using primal-dual hybrid gradient descent.
result Competitive performance in large-scale cooperative multi-agent environments.
ReF-ER algorithm improved performance in multi-agent reinforcement learning.
problem Improving performance in multi-agent reinforcement learning environments.
method Extended ReF-ER algorithm to include dependencies between agents and modeled environment dynamics.
result ReF-ER MARL outperforms state-of-the-art algorithms in collaborative environments.
Review of recent advances in decentralized MARL with networked agents.
problem Sequential decision-making by networked agents without central coordination.
method Analysis of recent algorithms and their theoretical foundations.
result Inspiring more research in this challenging area.
Bayesian network approach for efficient cooperative MARL.
problem Leveraging inter-agent coupling information for scalable MARL algorithms.
method Modeling cooperative MARL via Bayesian networks, identifying value dependency sets, proposing P-DTDE paradigm.
result P-DTDE policy gradient estimator has lower total variance than CTDE.
EMIX minimizes surprise in multi-agent reinforcement learning.
problem Surprise and approximation bias in multi-agent reinforcement learning.
method Energy-based MIXer (EMIX) for minimizing surprise across multiple agents.
result EMIX demonstrates consistent stable performance in challenging StarCraft II scenarios.
This work benchmarks MARL algorithms in cooperative tasks.
problem Lack of evaluation tasks and criteria for comparing MARL algorithms.
method Systematic evaluation of three MARL algorithm classes in diverse cooperative tasks.
result Insights into the effectiveness of different learning approaches.
Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action sp…
A method to train multi-agent reinforcement learning models without intrinsic rewards.
problem Training multi-agent reinforcement learning models with sparse rewards is challenging.
method A learning-based exploration strategy using variational graph autoencoder to generate initial states.
result The method improves the training and performance of multi-agent reinforcement learning models.
The recent success of single-agent reinforcement learning (RL) in Internet of things (IoT) systems motivates the study of multi-agent reinforcement learning (MARL), which is more challenging but more useful in large-scale IoT. In this paper, we consider a voting-based MARL problem, in which the agents vote to make grou…
DSAC improves cooperative MARL with general utilities, converging faster than existing methods.
problem Improving cooperation in multi-agent reinforcement learning with nonlinear utilities.
method Decentralized Shadow Reward Actor-Critic (DSAC) that estimates local occupancy measures and derivatives.
result DSAC converges to ε-stationarity in O(1/ε^2.5) steps with high probability, finding globally optimal policies.
Unified framework for randomized exploration in cooperative MARL.
problem Efficient exploration in cooperative multi-agent reinforcement learning.
method Unified algorithm framework with two Thompson Sampling algorithms, CoopTS-PHE and CoopTS-LMC.
result Theoretical O ~ ( d 3 / 2 H 2 M K ) \widetilde{\mathcal{O}}(d^{3/2}H^2\sqrt{MK}) O ( d 3/2 H 2 M K ) regret bound for parallel MDPs with linear transition. A scalable MARL algorithm using local rewards for cooperative multi-agent learning.
problem Scalability issues in cooperative multi-agent reinforcement learning due to large state and action spaces.
method LOMAQ algorithm incorporating local rewards in centralized training and decentralized execution.
result LOMAQ scales well compared to other methods, improving performance and convergence speed.
Attackers can significantly reduce team rewards in cooperative multi-agent reinforcement learning.
problem Robustness of cooperative multi-agent reinforcement learning to adversaries.
method Novel attack method involving training a policy network and using targeted adversarial examples.
result Reduces team reward from 20 to 9.4 by attacking a single agent, reducing winning rate from 98.9% to 0%.
New scalable MARL framework for dynamic networked systems.
problem Scalability in multi-agent reinforcement learning with dynamic dependencies.
method Scalable Actor Critic framework for non-local and stochastic dependencies.
result Finite-time error bound showing convergence rate dependence on information spread speed.
EOI enhances individuality in multi-agent systems.
problem Improving multi-agent cooperation through individuality.
method A probabilistic classifier that predicts agent identity based on observations, with intrinsic rewards and regularizers.
result EOI significantly outperforms existing methods in multi-agent cooperative scenarios.
This paper optimizes model-based RL for two-player zero-sum games with near-optimal sample complexity.
problem Optimizing model-based reinforcement learning for two-player zero-sum games with minimal samples.
method Model-based reinforcement learning approach for two-player discounted zero-sum Markov games with a generative model.
result Achieves a sample complexity of i l d e O ( ∣ S ∣ ∣ A ∣ ∣ B ∣ ( 1 − γ ) − 3 ε − 2 ) ilde O(|S||A||B|(1-γ)^{-3}ε^{-2}) i l d e O ( ∣ S ∣∣ A ∣∣ B ∣ ( 1 − γ ) − 3 ε − 2 ) for finding the Nash equilibrium and ε-NE policies. We study the problem of multi-agent reinforcement learning (MARL) with adaptivity constraints -- a new problem motivated by real-world applications where deployments of new policies are costly and the number of policy updates must be minimized. For two-player zero-sum Markov Games, we design a (policy) elimination base…
MARL algorithm uses regularization to avoid explicit structures, improving performance.
problem Lack of effective reinforcement learning methods for multi-agent systems.
method MARQ uses regularization to promote structured exploration without explicit centralized structures.
result MARQ outperforms existing methods in multi-agent environments.
EPC curriculum improves MARL performance as agent population grows.
problem Challenges in learning good policies for large multi-agent systems.
method Evolutionary Population Curriculum (EPC) for scaling MARL.
result EPC consistently outperforms baselines as agent population increases.
New MARL algorithms resolve the curse of multiagency with function approximation.
problem Challenges in Multi-Agent Reinforcement Learning (MARL) due to the curse of multiagency.
method V-Learning with Policy Replay and Decentralized Optimistic Policy Mirror Descent.
result First polynomial sample complexity results for learning approximate Coarse Correlated Equilibria (CCEs) of Markov Games under decentralized linear function approximation.
A new MARL framework for community-based cooperation with transfer and active exploration.
problem Flexible coordination patterns in multi-agent systems with community structures.
method Community-based multi-agent reinforcement learning with transfer and active exploration.
result Provably convergent actor-critic algorithms for structured information sharing and transfer learning.
Improved exploration in cooperative multi-agent reinforcement learning.
problem Limited expressiveness of Gaussian policies in DecSPG hinders effective exploration.
method Proposes decentralized diffusion policy learning (DDPL) with denoising diffusion probabilistic models.
result Consistently improved performance on various MARL benchmarks.
QPLEX learns efficient multi-agent Q-values by enforcing IGM principle.
problem Scalable multi-agent reinforcement learning with IGM consistency.
method Dueling duplex network architecture to enforce IGM principle.
result QPLEX achieves high sample efficiency and benefits from offline data.
Traffic congestion in metropolitan areas is a world-wide problem that can be ameliorated by traffic lights that respond dynamically to real-time conditions. Recent studies applying deep reinforcement learning (RL) to optimize single traffic lights have shown significant improvement over conventional control. However, o…
Paper tackles RL for power grid topology optimization.
problem Managing large action spaces in growing power networks.
method Hierarchical multi-agent reinforcement learning (MARL) framework.
result MARL framework outperforms single-agent RL methods.
Finding the optimal signal timing strategy is a difficult task for the problem of large-scale traffic signal control (TSC). Multi-Agent Reinforcement Learning (MARL) is a promising method to solve this problem. However, there is still room for improvement in extending to large-scale problems and modeling the behaviors …
New algorithm breaks multiagency gap in robust MARL.
problem Vulnerability of MARL to sim-to-real gaps.
method Distributionally robust Markov games (RMGs) with a new uncertainty set formulation.
result First algorithm to break the curse of multiagency for RMGs.