Algorithm learns Nash equilibria in stochastic games using entropy-regularized policies.
problem Learning Nash equilibria in zero-sum stochastic games is computationally expensive.
method Entropy-regularized soft policies for Q-function updates.
result Algorithm converges to Nash equilibrium under certain conditions.
Policy gradient methods find Nash equilibrium in noisy games.
problem Finding Nash equilibrium in noisy games.
method Policy gradient methods with noise added.
result Policy gradient methods converge to Nash equilibrium in noisy games.
A new method for RLHF using proximal point Nash learning.
problem Capturing real human preferences in RLHF.
method Proximal point Nash learning, embedding self-play updates into a proximal point framework.
result High-probability last-iterate convergence for the combined method.
A RL approach finds Nash equilibrium for turn-based zero-sum games.
problem Finding Nash equilibrium in two-player turn-based zero-sum games.
method EIS method combining exploration, policy improvement, and supervised learning.
result EIS method finds an ε-approximate value function of Nash equilibrium in O(ε^(-(d+4))) steps.
A new algorithm reduces memory and computational needs for reinforcement learning.
problem Memory and computational inefficiency in model-free reinforcement learning.
method Memory-Efficient Nash Q-Learning (ME-Nash-QL) for two-player zero-sum games.
result Proves ME-Nash-QL reduces space and sample complexity for tabular and long-horizon cases.
Paper shows equivalence between two alignment methods and introduces a new algorithm.
problem Ensuring human alignment of large language models for useful, safe, and pleasant user experience.
method Introduces IPO-MD algorithm, showing equivalence between IPO and Nash-MD methods.
result Equivalence between IPO and Nash-MD methods proven when considering online version of IPO.
A new approach to fine-tuning LLMs with human feedback.
problem Inability of current reward models to fully represent human preferences.
method Introducing NLHF, a new pipeline for LLM fine-tuning using pairwise human feedback.
result NLHF produces a sequence of policies converging to the regularized Nash equilibrium.
Policy mirror ascent achieves Nash equilibrium in mean field games without a population generative model.
problem Achieving Nash equilibrium in mean field games without a population generative model.
method Policy mirror ascent, contractive operator, single-path TD learning.
result Policy mirror ascent converges to Nash equilibrium within O ~ ( ε − 2 ) \widetilde{\mathcal{O}}(\varepsilon^{-2}) O ( ε − 2 ) samples. We show by counterexample that policy-gradient algorithms have no guarantees of even local convergence to Nash equilibria in continuous action and state space multi-agent settings. To do so, we analyze gradient-play in N-player general-sum linear quadratic games, a classic game setting which is recently emerging as a b…
We study the global convergence of policy optimization for finding the Nash equilibria (NE) in zero-sum linear quadratic (LQ) games. To this end, we first investigate the landscape of LQ games, viewing it as a nonconvex-nonconcave saddle-point problem in the policy space. Specifically, we show that despite its nonconve…
New algorithm reduces online learning regret in uninformed Markov games.
problem Achieving no external regret in uninformed Markov games is impossible.
method Empirical Nash-value regret, parameter-free algorithm, adaptive restart.
result Achieves O ( min { K + ( C K ) 1 / 3 , L K } ) O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\}) O ( min { K + ( C K ) 1/3 , L K }) regret bound. Algorithm converges to Nash equilibria in competitive games.
problem Finding Nash equilibria in decentralized, competitive Markov games.
method Decentralized Optimistic Gradient Descent/Ascent with a critic.
result Converges to the set of Nash equilibria under self-play.
Dynamic pricing policy converges to Nash equilibrium with low regret.
problem Sequential price competition among sellers over multiple periods.
method Semi-parametric least-squares estimation of s-concave demand functions.
result Prices converge to Nash equilibrium with rate O ( T − 1 / 7 ) O(T^{-1/7}) O ( T − 1/7 ) and sellers incur regret O ( T 5 / 7 ) O(T^{5/7}) O ( T 5/7 ) . Policy gradient method proves convergence in imperfect-information games.
problem Policy gradient methods in imperfect-information games (EFGs).
method Policy gradient approach with best-iterate convergence.
result Policy gradient leads to provable best-iterate convergence in self-play EFGs.
Study proposes new OPE estimators for two-player zero-sum games.
problem Evaluating new policies using historical data from a different policy in multi-player zero-sum games.
method Doubly robust and double reinforcement learning estimators to project exploitability.
result Prove exploitability estimation error bounds and regret bounds for policy profiles.
New RL algorithms find SNE in Markov games with myopic followers.
problem Finding SNE in Markov games with myopic followers.
method Optimistic and pessimistic variants of least-squares value iteration, incorporating function approximation.
result First provably efficient RL algorithms for SNEs in general-sum Markov games with myopic followers.
MF-TRPO optimizes MFGs with finite sample guarantees.
problem Computing approximate Nash equilibria in MFGs.
method Extends TRPO to MFGs, providing convergence guarantees.
result Theoretical guarantees on MF-TRPO's convergence.
We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean o…
A new approach for cooperative multi-agent reinforcement learning with limited communication, reducing the number of communication rounds.
problem Limited communication in decentralized MARL systems leads to outdated information and unstable learning.
method Base policy prediction technique to estimate gradients and collect samples for a sequence of base policies.
result The proposed algorithm converges to an ε-Nash equilibrium with significantly fewer communication rounds and samples.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
New algorithm finds Nash equilibrium in multi-agent games.
problem Extending RL theory to multi-agent settings with large state spaces.
method Proposes an algorithm using an exploiter to find Nash equilibrium policies.
result Proves finding Nash equilibrium is possible with polynomial samples.
Two firms compete in a financial market, choosing dividend strategies to avoid default and maximize profits.
problem Strategic interaction between two financially constrained firms in a market with default risk.
method Construct Nash equilibria in feedback form for a class of two-person stochastic games of singular control.
result Explicit identification of optimal strategies and equilibrium payoffs for different initial conditions.
A method to learn robust policies for environments with model mismatches.
problem Training agents in high-stakes scenarios with mismatched training and real environments.
method Formalizes the perturbation as a zero-sum game to find Nash Equilibrium, which corresponds to the robust policy.
result Our algorithm can find a near-optimal robust policy with high probability using polynomial samples.
New algorithms converge faster to Nash equilibrium in zero-sum games with bandit feedback.
problem Learning in zero-sum games with bandit feedback without communication.
method Developed two uncoupled algorithms achieving optimal rate of Ω ( T − 1 / 4 ) Ω(T^{-1/4}) Ω ( T − 1/4 ) . result Achieved optimal rate of Ω ( T − 1 / 4 ) Ω(T^{-1/4}) Ω ( T − 1/4 ) for convergence of policy profiles to Nash equilibrium. Paper optimizes reinforcement learning in self-play games with reduced steps.
problem Optimizing reinforcement learning algorithms for self-play in two-player zero-sum games.
method Proposes optimistic Nash Q-learning and Nash V-learning algorithms with improved sample complexity.
result Achieves sample complexity of O ( S A B ) O(SAB) O ( S A B ) for Nash Q-learning and O ( S ( A + B ) ) O(S(A+B)) O ( S ( A + B )) for Nash V-learning, closing the gap with lower bounds. Decentralized algorithm reduces regret and converges to Nash equilibrium in online congestion games.
problem Online congestion games with exponential action sets and strict Nash equilibria.
method CongestEXP algorithm using exponential weights method.
result CongestEXP achieves O ( k F T ) O(kF\sqrt{T}) O ( k F T ) regret bound and almost exponential convergence to strict Nash equilibrium. We consider the problem of finding stationary Nash equilibria (NE) in a finite discounted general-sum stochastic game. We first generalize a non-linear optimization problem from Filar and Vrieze [2004] to a N N N -player setting and break down this problem into simpler sub-problems that ensure there is no Bellman error fo…
Paper presents a GMFG framework for large stochastic games.
problem Learning Nash Equilibrium in large stochastic games.
method Value-based and policy-based reinforcement learning algorithms with smoothed policies.
result Proposed algorithms GMF-V and GMF-P are efficient and robust in GMFG setting.
New algorithm improves self-play reinforcement learning for competitive games.
problem Inefficient opponent selection in self-play reinforcement learning.
method Intelligently selects opponents based on adversarial rules derived from saddle point optimization.
result Algorithm converges to approximate equilibrium with high probability in convex-concave games.
We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…
NOHD optimizes multi-agent systems by decomposing dynamics into irrotational and solenoidal components.
problem Non-stationarity and conflicting interests in multi-agent learning problems.
method NOHD (Newton Optimization on Helmholtz Decomposition) decomposes system dynamics into irrotational and solenoidal components.
result NOHD ensures quadratic convergence in purely irrotational and solenoidal systems and attracts to stable fixed points in general multi-agent systems.
Improved model-based reinforcement learning for multi-agent Markov games.
problem Suboptimal sample complexity for model-based algorithms in multi-agent reinforcement learning.
method Optimistic Nash Value Iteration (Nash-VI) for two-player zero-sum Markov games.
result First model-based algorithm matching information-theoretic lower bound with improved sample complexity.
Study efficient offline RL in Markov games with general models.
problem Learn approximate equilibria from offline data in Markov games.
method Use Bellman-consistent pessimism for interval estimation and optimize gap relaxation.
result First framework for sample-efficient offline learning in Markov games, handling all equilibria.
The paper tackles fair sharing of exploration costs across groups in online learning.
problem Sharing the cost of exploration fairly across multiple groups in online learning.
method The paper introduces the 'grouped' bandit model and uses axiomatic bargaining theory, specifically the Nash bargaining solution, to formalize fairness.
result The paper derives policies that are optimally fair and regret-optimal, showing that regret-optimal policies can be unfair.
The paper explores game-theoretic alignment of LLMs with human preferences, finding limitations and conditions.
problem Aligning LLMs with human preferences using game theory.
method Systematic study of payoff choices in a two-player zero-sum game for desirable alignment properties.
result Impossibility of preference matching in game-theoretic LLM alignment under standard assumptions.
This paper simplifies complex game dynamics by using a recursive representation.
problem Difficulties in finite-player dynamic games with private information.
method Provides a recursive representation and noise-state model.
result Equilibrium becomes a deterministic fixed point in impulse-response functions.
Game-theoretic analysis of mining gaps in blockchain systems.
problem Strategic mining behavior and its impact on blockchain stability.
method Game-theoretic model and Nash equilibrium analysis.
result Mining gaps can destabilize blockchain systems, especially with decreasing block rewards.
Novel algorithms for multi-agent reinforcement learning reduce sample complexity.
problem Efficiently learning Nash equilibria in multi-agent settings.
method Information-Directed Sampling (IDS) principles applied to multi-agent reinforcement learning.
result Sample-efficient algorithms for learning Nash equilibria in various multi-agent settings.
Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradi…
The paper solves TIC LQ control problems using stochastic differential games.
problem Time-inconsistent linear-quadratic stochastic control problems.
method Stochastic differential games, spike variation approach.
result Achieves Nash equilibrium for TIC problems, demonstrating impact of ambiguity aversion.
Let M M M and N N N be Nash manifolds, and f f f and g g g Nash maps from M M M to N N N . If M M M and N N N are compact and if f f f and g g g are analytically R-L equivalent, then they are Nash R-L equivalent. In the local case, C i n f t y C^infty C i n f t y R-L equivalence of two Nash map germs implies Nash R-L equivalence. This shows a difference of Nash…
Models analyze strategic risk-taking in continuous action games.
problem Strategic risk-taking dynamics in continuous action games.
method Normal form game, multi-player scenarios, regret minimization algorithms, numerical algorithm for calculation.
result Nash equilibrium also serves as a correlated equilibrium in continuous games.
UBI model proves financial equilibrium exists.
problem Proving existence of financial equilibrium with UBI.
method Backward stochastic differential equation (BSDE) approach.
result Equilibrium exists in UBI model.
The paper analyzes a game where players must balance short-term and long-term interests, leading to cooperative or competitive outcomes.
problem Analyzing time inconsistency in inter-personal decision-making under non-exponential discounting.
method Iterative procedures and Zorn's lemma to find Nash equilibria between players' intra-personal equilibria.
result Inter-personal equilibria exist and depend on the impatience levels of the players.
Nash's theorem proved with Günther's trick
problem Proving Nash's smooth embedding theorem
method Using Günther's trick
result Nash's theorem proved
SPPO optimizes language model alignment by treating preferences as a game and achieving state-of-the-art performance.
problem Capturing intransitivity and irrationality in human preferences for accurate language model alignment.
method Self-play-based approach to identify Nash equilibrium policy through iterative policy updates.
result SPPO achieves state-of-the-art win-rate of 28.53% on AlpacaEval 2.0 without external supervision.
Study optimizes rebate design in auction markets to enhance efficiency.
problem Designing optimal rebate policies in auction markets to improve efficiency.
method Formulated as a principal-agent problem, solved using Hamilton-Jacobi-Bellman equations and Deep BSDE method.
result Optimal transaction fees and rebates narrow the price spread, improving market efficiency.
We propose local symplectic surgery, a two-timescale procedure for finding local Nash equilibria in two-player zero-sum games. We first show that previous gradient-based algorithms cannot guarantee convergence to local Nash equilibria due to the existence of non-Nash stationary points. By taking advantage of the differ…