Study online learning in unknown Markov games with sublinear regret.
problem Online learning in unknown Markov games with unobservable opponents.
method Introduced an algorithm achieving sublinear regret against the minimax value.
result First sublinear regret bound for unknown Markov games, independent of action spaces size.
New RL algorithms find SNE in Markov games with myopic followers.
problem Finding SNE in Markov games with myopic followers.
method Optimistic and pessimistic variants of least-squares value iteration, incorporating function approximation.
result First provably efficient RL algorithms for SNEs in general-sum Markov games with myopic followers.
Algorithm finds Nash equilibria in complex games with function approximation.
problem Learning Nash equilibria in two-player zero-sum Markov Games with nonlinear function approximation.
method Online learning algorithm using upper and lower confidence bounds derived from optimism in the face of uncertainty.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret with polynomial complexity, under mild assumptions. Algorithm minimizes regret and converges to equilibria in Markov games.
problem Regret minimization and convergence to equilibria in general-sum Markov games under adversarial opponents.
method Decentralized algorithm that uses policy optimization and controls path length to achieve sublinear regret.
result Sublinear regret guarantees for convergence to correlated equilibrium in Markov games.
Efficient reinforcement learning for simultaneous-move zero-sum games using optimistic value iteration.
problem Learning optimal strategies in simultaneous-move zero-sum Markov games with function approximation.
method Developed an optimistic variant of least-squares minimax value iteration algorithm for offline and online settings.
result Achieved an upper bound of i l d e O ( d 3 H 3 T ) ilde O(\sqrt{d^3 H^3 T}) i l d e O ( d 3 H 3 T ) on duality gap and regret. New research shows no-regret learning is impossible in Markov games under certain assumptions.
problem Achieving no-regret learning in decentralized Markov games.
method Novel application of aggregation techniques from online learning to prove lower bounds.
result No polynomial-time algorithm exists for independent no-regret learning in general-sum Markov games.
Algorithm learns robust equilibrium in online Markov games with interactive data.
problem Sim-to-real gap in reinforcement learning.
method Distributionally robust RL with minimum value assumption, least square value iteration.
result Sample-efficient algorithm for robust equilibrium in online Markov games.
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
Paper optimizes multi-agent learning in Markov games with generative model.
problem Learning Nash or CCE equilibria in multi-agent Markov games.
method Develops \myalg~algorithm and adaptive sampling scheme using FTRL method.
result Minimax-optimal learning of CCE with minimal samples.
Algorithm learns NE in imperfect information games with imperfect feedback.
problem Learning Nash equilibrium in imperfect information games with bandit feedback.
method IXOMD algorithm for model-free learning with 1 / T 1/\sqrt{T} 1/ T convergence rate. result IXOMD achieves 1 / T 1/\sqrt{T} 1/ T convergence rate to NE. New algorithm reduces online learning regret in uninformed Markov games.
problem Achieving no external regret in uninformed Markov games is impossible.
method Empirical Nash-value regret, parameter-free algorithm, adaptive restart.
result Achieves O ( min { K + ( C K ) 1 / 3 , L K } ) O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\}) O ( min { K + ( C K ) 1/3 , L K }) regret bound. DORIS algorithm achieves no-regret learning in Markov games with adversarial opponents.
problem Decentralized policy learning in Markov games with nonstationary opponents.
method DORIS algorithm using optimistic hyperpolicy mirror descent.
result Achieves K \sqrt{K} K -regret in general function approximation. New algorithms for RL in Markov games with independent linear function approximation, breaking the curse of multiagents.
problem Tackles the challenge of learning Markov equilibria in large state space Markov games with multiple agents.
method Proposes independent linear Markov games and designs new algorithms for learning Markov coarse correlated equilibria and Markov correlated equilibria with polynomial sample complexity.
result Breaks the curse of multiagents by achieving sample complexity bounds that scale polynomially with each agent's function class complexity.
New assumptions and algorithm solve offline two-player zero-sum Markov games.
problem Solving offline two-player zero-sum Markov games under insufficient assumptions.
method Proposed unilateral concentration assumption and pessimism-type algorithm.
result Algorithm efficiently learns Nash equilibrium under unilateral concentration.
New algorithm finds near-optimal policies efficiently in zero-sum games.
problem Lack of provable efficiency guarantees for policy optimization in zero-sum games.
method Policy optimization algorithm with function approximation.
result Proves efficient convergence to near-optimal policies with polynomial samples and iterations.
Maximize to Explore integrates RL components for efficient policy discovery.
problem Balancing exploration and exploitation in online RL with general function approximators.
method Integrates estimation, planning, and exploration into a single objective function.
result Achieves sublinear regret for MDPs and MGs with general function approximations.
Improved model-based reinforcement learning for multi-agent Markov games.
problem Suboptimal sample complexity for model-based algorithms in multi-agent reinforcement learning.
method Optimistic Nash Value Iteration (Nash-VI) for two-player zero-sum Markov games.
result First model-based algorithm matching information-theoretic lower bound with improved sample complexity.
In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power co…
Smart grid uses deep learning to optimize household energy use.
problem Optimizing household energy use under real-time pricing schemes.
method Multi-agent deep actor-critic learning for decentralized agents with partial observability.
result Deep reinforcement learning reduces peak-to-average energy consumption and costs.
We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as t…
New algorithms achieve logarithmic regret in KL-regularized Markov games.
problem Improving sample efficiency in game-theoretic settings with KL regularization.
method Developed OMG and SOMG algorithms for matrix and Markov games, using best response sampling and superoptimistic bonuses.
result Logarithmic regret in T T T that scales inversely with KL regularization strength β β β . The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independe…
Algorithm finds ε-equilibrium policies for multi-agent Markov games with hidden low-rank structure.
problem Designing efficient algorithms for multi-agent Markov games with unknown representation and hidden low-rank structure.
method Model-based and model-free approaches using representation learning to construct an effective representation from data.
result Achieves poly ( H , d , A , 1 / ε ) (H,d,A,1/\varepsilon) ( H , d , A , 1/ ε ) sample complexity for both model-based and model-free approaches. The paper simplifies multi-agent RL dynamics in finite-state Markov games using homogenization.
problem Approximating complex multi-agent reinforcement learning dynamics in finite-state Markov games.
method Rescaling learning process by reducing learning rate and increasing update frequency, proving convergence to an ODE.
result The rescaled process converges to an ODE that approximates the agent's learning dynamics.
Paper develops efficient algorithms for zero-sum Markov games with general function classes.
problem Challenging settings in zero-sum Markov games with parameterized value functions or models.
method Developed new model-free and model-based algorithms for decoupled and coordinated settings.
result Improved sample complexity and regret bounds for various settings.
Algorithm solves online binary classification and infinite games using ERM oracle.
problem Online learning and solving infinite games with computationally inefficient oracles.
method Proposes an algorithm relying solely on ERM oracle calls for online binary classification and nonparametric games.
result Achieves finite and sublinearly growing regret in various settings.
Study efficient offline RL in Markov games with general models.
problem Learn approximate equilibria from offline data in Markov games.
method Use Bellman-consistent pessimism for interval estimation and optimize gap relaxation.
result First framework for sample-efficient offline learning in Markov games, handling all equilibria.
A method to learn robust policies for environments with model mismatches.
problem Training agents in high-stakes scenarios with mismatched training and real environments.
method Formalizes the perturbation as a zero-sum game to find Nash Equilibrium, which corresponds to the robust policy.
result Our algorithm can find a near-optimal robust policy with high probability using polynomial samples.
We introduce CSE for MLSF games and devise online learning algorithms for achieving no-external Stackelberg-regret.
problem Learning equilibrium in leader-follower games with noisy bandit feedback.
method Proposed Correlated Stackelberg Equilibrium (CSE) and online learning algorithms balancing exploration and exploitation.
result Achieves no-external Stackelberg-regret, converging to approximate CSE.
New RL algorithm tackles online robust MDPs with uncertainty.
problem Developing robust reinforcement learning models for real-world environments.
method Proposes a robust optimistic policy optimization algorithm for online robust MDPs.
result Establishes the first regret bound for online robust MDPs.
Optimal algorithm for two-player zero-sum games with linear parameterization.
problem Finding Nash Equilibrium in two-player zero-sum Markov games with linear transition.
method Nash-UCRL algorithm, Coarse Correlated Equilibrium, Optimism-in-Face-of-Uncertainty.
result Proves i l d e O ( d H T ) ilde{O}(dH\sqrt{T}) i l d e O ( d H T ) regret bound, matching lower bound up to logarithmic factors. Pessimistic model-based algorithm finds Nash equilibria in zero-sum Markov games from offline data.
problem Learning Nash equilibria in two-player zero-sum Markov games from limited data.
method Pessimistic model-based algorithm with Bernstein-style lower confidence bounds (VI-LCB-Game).
result Proves sample complexity no larger than C c l i p p e d ⋆ S ( A + B ) ( 1 − γ ) 3 ε 2 \frac{C_{\mathsf{clipped}}^\star S(A+B)}{(1-γ)^3 \varepsilon^2} ( 1 − γ ) 3 ε 2 C clipped ⋆ S ( A + B ) , achieving minimax optimality. The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.
problem Analyzing the cumulative regret of Nash Q-learning in 2-player turn-based stochastic Markov games.
method Proposed gap-dependent logarithmic upper bounds for cumulative regret in episodic tabular setting and discounted game setting.
result The proposed bounds match theoretical lower bounds up to a logarithmic term.
New algorithms achieve near-optimal cumulative loss in nonparametric online learning and games.
problem Fast rates of convergence in nonparametric online regression and classification.
method Randomized proper learning algorithms, hierarchical aggregation, multi-scale extension, stability proof.
result Achieved near-optimal cumulative loss bounds for real-valued and binary games.
Contributions: Prior studies on education have mostly followed the model of the cross sectional study, namely, examining the pretest and the posttest scores. This paper shows that students' knowledge throughout the intervention can be estimated by time series analysis using a hidden Markov model. Background: Analyzing …
This work tackles learning Markov games with adversarial opponents and achieves both average reward and exploitation.
problem Achieving both average reward and exploiting adaptive opponents in Markov games.
method Develops efficient algorithms and proves hardness results for learning Markov games with adversarial opponents.
result Achieves K \sqrt{K} K -regret bounds for certain conditions on opponent policies, complemented by an exponential lower bound. A new method solves bilevel optimization problems in competitive Markov games.
problem Capturing competitive structures in RL with multiple interacting policies.
method Penalty-augmented Nikaido-Isoda descent-ascent (PANDA) method.
result PANDA converges to stationary points without convexity assumptions.
Study on learning strategies in adaptive Markov games with policy regret as metric.
problem Learning in dynamic Markov games with adaptive opponents is challenging.
method Introduced policy regret as a new learning metric and developed algorithms for consistent adaptive adversaries.
result Achieved T \sqrt{T} T policy regret against certain adaptive adversaries. Algorithm improves RL model selection for repeated games with utility maximization.
problem Optimal policy learning in repeated games with unknown opponent strategy.
method Proposes MRBEAR for average reward RL, applying to utility maximization in repeated games.
result Regret bound shows linear dependence on number of model classes in average reward RL.
New framework connects online learning to statistical learning for better generalization bounds.
problem Deriving generalization bounds for statistical learning algorithms.
method Constructing an online learning game and showing a connection to statistical learning.
result Established a connection between online and statistical learning, leading to new generalization bounds.
New algorithm improves sample efficiency for zero-sum Markov games.
problem Improving sample efficiency for model-free algorithms in zero-sum Markov games.
method Proposes a model-free stage-based Q-learning algorithm using variance reduction techniques.
result Achieves optimal sample complexity for finding ε-optimal Nash Equilibrium.
This paper improves sample efficiency for learning equilibria in multi-player games.
problem Sample-efficient learning of equilibria in games with many players.
method Designs algorithms for learning CCE and CE with polynomial sample complexity in the number of players.
result First to show polynomial sample complexity for learning CCE and CE in multi-player games.
New game design method for better trait inference.
problem Inferring latent psychological traits from human behavior.
method Formulated as a mutual information maximization problem, solved using variational lower bound optimization.
result Designed games successfully distinguish among players with different traits, outperforming traditional methods.
The paper extends game theory using Hodge theory on graphs.
problem Generalizing Shapley's value allocation formula for cooperative games on graphs.
method Connecting stochastic path integrals to Hodge-theoretic Poisson's equations on graphs.
result The value allocation operator is the solution to Poisson's equation in combinatorial Hodge theory.
Decentralized algorithm reduces regret and converges to Nash equilibrium in online congestion games.
problem Online congestion games with exponential action sets and strict Nash equilibria.
method CongestEXP algorithm using exponential weights method.
result CongestEXP achieves O ( k F T ) O(kF\sqrt{T}) O ( k F T ) regret bound and almost exponential convergence to strict Nash equilibrium. Paper analyzes convergence rates for multi-agent learning in games.
problem Convergence rates for multi-agent learning in games.
method Characterizes finite-time convergence rates for joint OGD learning on λ λ λ -cocoercive games and develops adaptive algorithms. result Adaptive algorithms achieve same convergence rates as non-adaptive counterparts.
This paper tackles sample-efficient reinforcement learning for partially observable Markov games.
problem Learning in partially observable Markov games with incomplete information.
method A simple algorithm combining optimism and Maximum Likelihood Estimation (MLE) for self-play, and a variant of optimistic MLE for adversarial opponents.
result The proposed algorithms achieve approximate Nash, correlated, and coarse correlated equilibria in polynomial samples for weakly revealing POMGs.
Study optimal policy regret in partially observable Markov games with adaptive opponents.
problem Optimal sequential decision-making in partially observable environments against strategic, adaptive opponents.
method An epoch-based optimistic maximum-likelihood algorithm that selects one policy per epoch using confidence sets built cumulatively from past data.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) policy regret for fixed problem parameters, with explicit dependence on horizon, adversary memory, confidence radius, and aggregate Eluder dimension.