New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
New RL algorithms achieve optimal policies with polynomial sample complexity for mean-field problems.
problem Statistical efficiency of Mean-Field Reinforcement Learning with general function approximation.
method Introduce MF-MBED to characterize problem complexity, propose algorithms based on maximal likelihood estimation.
result Rich mean-field RL problems have low MF-MBED, leading to polynomial sample complexity.
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.
Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.
problem Continuous-time mean-field control in reinforcement learning.
method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.
result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.
New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.
problem Solving mean-field control problems in continuous time reinforcement learning.
method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.
result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.
Paper presents a GMFG framework for large stochastic games.
problem Learning Nash Equilibrium in large stochastic games.
method Value-based and policy-based reinforcement learning algorithms with smoothed policies.
result Proposed algorithms GMF-V and GMF-P are efficient and robust in GMFG setting.
Study policy gradient for large-agent mean-field control and game in continuous time.
problem Optimal policy learning for large number of agents in continuous-time mean-field systems.
method Policy gradient method applied to linear-quadratic mean-field control and game models.
result Policy gradient converges to optimal solution at a linear rate for both mean-field control and game.
Entropy regularization improves MFG learning efficiency and stability.
problem Improving Mean Field Game learning efficiency and stability.
method Entropy regularization applied to MFG with learning.
result Entropy regularization yields time-dependent policies and stabilizes convergence.
Safe-M3-UCRL learns safe policies for multi-agent systems with global constraints.
problem Global constraints in mean-field reinforcement learning for multi-agent systems.
method Safe-M3-UCRL uses epistemic uncertainty and log-barrier approach to ensure constraints satisfaction. result Safe-M3-UCRL learns safe policies for multi-agent systems with global constraints. MF-TRPO optimizes MFGs with finite sample guarantees.
problem Computing approximate Nash equilibria in MFGs.
method Extends TRPO to MFGs, providing convergence guarantees.
result Theoretical guarantees on MF-TRPO's convergence.
Efficiently learns MFC systems with unknown dynamics.
problem Learning in multi-agent systems with non-stationary interactions and combinatorial state/action spaces.
method Model-based reinforcement learning algorithm, M3−UCRL, balancing exploration and exploitation. result First general regret bounds for model-based reinforcement learning of MFC systems.
The paper tackles safe reinforcement learning with convex regularization.
problem Safe reinforcement learning in complex, high-dimensional settings with safety constraints.
method Doubly-regularized RL framework combining reward and parameter regularization, formulated as a convex regularized objective with parametrized policies on an infinite-dimensional statistical manifold.
result Exponential convergence guarantees under sufficient regularization, robust theoretical insights and guarantees for safe RL.
Study on PG learning for LQ MFC problems with common noise, proving convergence and sample complexity.
problem Optimal policy learning in LQ MFC problems with common noise and entropy regularization.
method Comprehensive error analysis of PG algorithms in both model-based and model-free settings.
result Global linear convergence and sample complexity of PG algorithms in model-free setting.
Two new RL methods enable deep learning of MFG equilibria.
problem Efficiently learning equilibria in Mean Field Games using RL.
method Two novel RL methods: distillation and online mixing.
result Deep RL algorithms can now solve various MFGs.
Develops a new reinforcement learning framework for complex control problems.
problem Continuous-time extended mean field control with deterministic policies.
method Model-free sensitivity formula, deterministic policy gradient, local value and advantage-rate representations.
result Demonstrates efficiency, stability, and robustness in solving complex control problems.
RL in MFGs is as hard as solving many single-agent RL problems.
problem Learning Nash Equilibrium in Mean-Field Games (MFGs).
method Introduce P-MBED to measure model complexity, develop a novel exploration strategy, and establish polynomial sample complexity results.
result Learning Nash Equilibrium in MFGs is no more statistically challenging than solving a logarithmic number of single-agent RL problems.
This paper develops q-learning methods for mean-field control problems.
problem Continuous-time mean-field control problems with interaction between agents.
method Introduces two q-functions and devises model-free learning algorithms.
result Developed algorithms can learn optimal value functions and q-functions.
Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously, the state of every agent moves to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often …
New algorithms learn graphons in GMFGs without knowing them.
problem Learning graphons in Graphon Mean-Field Games with unknown graphons.
method Proximal Policy Optimization for GMFG (GMFG-PPO) and kernel embedding methods for estimating graphons.
result The proposed algorithms reduce exploitability when learning unknown graphons.
DeepGSB solves MFGs with non-differentiable preferences.
problem Solving MFGs with non-differentiable preferences and exact population convergence.
method Generalized Schrödinger Bridge via Forward-Backward SDEs and Temporal Difference learning.
result DeepGSB provides necessary and sufficient conditions for mean-field problems.
We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean o…
In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power co…
Paper develops MFC approach to reduce MARL complexity.
problem Curse of dimensionality in cooperative MARL.
method Mean-field control (MFC) and Q-learning.
result Linear convergence rate for MFC-K-Q, independent of N. Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on…
Expands MFGs to handle real-world asymmetric multi-agent games efficiently.
problem Applying mean-field games to real-world, heterogeneous multi-agent systems.
method Develops a method to symmetrize and extend finite-player games to infinite-player MFGs, proving approximation bounds and convergence guarantees.
result TD learning converges to approximate Nash equilibria in finite-sample settings, enabling symmetrized learning without explicit MFG models.
We propose a general framework for solving statistical mechanics of systems with finite size. The approach extends the celebrated variational mean-field approaches using autoregressive neural networks, which support direct sampling and exact calculation of normalized probability of configurations. It computes variation…
Abstract: Surveying connections between ML and Control Theory.
problem Addressing the intersection of Machine Learning and Control Theory.
method Develops connections through reinforcement learning, supervised learning, deep learning, and stochastic gradient descent.
result Machine Learning and Control Theory are interconnected, with ML solving large control problems and Control Theory providing tools for ML.
Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number o…
Unified q-learning for mean-field jump-diffusion models with unobservable population distribution.
problem Continuous-time q-learning in mean-field jump-diffusion models with unobservable population distribution.
method Proposed decoupled Iq-function for unified policy evaluation in MFG and MFC problems; unified q-learning algorithm based on test policies and averaged martingale orthogonality condition.
result Unified policy evaluation rule for MFG and MFC problems based on decoupled Iq-function.
Stochastic approximation (SA) is a key method used in statistical learning. Recently, its non-asymptotic convergence analysis has been considered in many papers. However, most of the prior analyses are made under restrictive assumptions such as unbiased gradient estimates and convex objective function, which significan…
Prior distributions of binarized natural images are learned by using a Boltzmann machine. According the results of this study, there emerges a structure with two sublattices in the interactions, and the nearest-neighbor and next-nearest-neighbor interactions correspondingly take two discriminative values, which reflect…
Temporal-difference and Q-learning learn feature representations that converge to optimal ones.
problem Understanding how feature representations evolve in temporal-difference and Q-learning with neural networks.
method Mean-field theory applied to overparameterized two-layer neural networks.
result The feature representation converges to the optimal one, generalizing previous results.
New neural networks learn mappings between probability measures and functions.
problem Learning mappings between Wasserstein space of probability measures and function spaces.
method Two types of neural networks: bin density and cylindrical approximation, are proposed and supported by universal approximation theorems.
result Accuracy and efficiency of mean-field neural networks in generalization error with various test distributions.
Federated learning linked to mean-field games for large-scale learning.
problem Large-scale distributed and privacy-preserving learning algorithms.
method Established a connection between federated learning and mean-field games, presenting federated learning as a differential game.
result Properties of the equilibrium of the federated learning game were discussed.
In this paper we study iterative procedures for stationary equilibria in games with large number of players. Most of learning algorithms for games with continuous action spaces are limited to strict contraction best reply maps in which the Banach-Picard iteration converges with geometrical convergence rate. When the be…
Study on kernel methods in large-scale machine learning problems.
problem Large-scale machine learning with many interacting variables.
method Mean field limit analysis of kernels and their Hilbert spaces.
result Mean field convergence of empirical and infinite-sample solutions.
Deep learning enhances solving complex mean field games in finance.
problem Solving large-scale mean field games with financial applications.
method Combining mean field games theory with deep learning techniques.
result Improved solutions for large-scale financial games.
Unified analysis of neural networks in NPIV using 2SLS and MFLD.
problem Global convergence of neural networks in NPIV.
method Lifted perspective through MFLD, penalty gradient approach for bilevel optimization.
result First global convergence result of neural networks for 2SLS in NPIV.
Elman-type RNNs converge to globally optimal solutions in the mean-field regime.
problem Optimizing feature learning in wide RNNs.
method Analysis of gradient descent dynamics and mean-field limits.
result Fixed points of infinite-width dynamics are globally optimal.
The mean field algorithm is a widely used approximate inference algorithm for graphical models whose exact inference is intractable. In each iteration of mean field, the approximate marginals for each variable are updated by getting information from the neighbors. This process can be equivalently converted into a feedf…
Robust Q-learning for mean-field control under Wasserstein uncertainty
problem Mean-field control under Wasserstein uncertainty
method Quantization-and-projection scheme with Wasserstein dual reformulation
result Convergence and finite-time iteration bounds
New methods learn correlated equilibria in large games without structural assumptions.
problem Learning correlated equilibria in large, anonymous games with exponential player count.
method Developed Mean-Field correlated and coarse-correlated equilibria, and used classical algorithms to learn them efficiently.
result Efficiently learned correlated equilibria in all games without structural assumptions.
Kernel methods are studied in a mean field limit for high-dimensional data.
problem Analyzing kernel methods in high-dimensional data with many variables.
method Investigation of kernel methods in the mean field limit of interacting particle systems.
result Rigorous mean field limit of kernels and detailed analysis of the limiting reproducing kernel Hilbert space.
A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
Paper studies convergence of Mean-Field GDA dynamics for MNE of continuous games.
problem Finding mixed Nash equilibria in continuous games.
method Two-scale Mean-Field Gradient Descent Ascent dynamics.
result Two-scale Mean-Field GDA converges exponentially to MNE without convexity assumptions.
Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.
problem Feature learning ability of two-layer neural networks in the mean-field regime.
method Mean-field analysis through kernel methods, focusing on dynamics of the first layer's kernel.
result Two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than kernel methods.
New algorithm reduces MFGs with common noise complexity.
problem Prohibitive computational cost in solving MFGs with common noise.
method Signatured deep fictitious play based on rough path theory.
result Significantly reduced computational complexity and improved efficiency.
Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to red…