RL in MFGs is as hard as solving many single-agent RL problems.
problem Learning Nash Equilibrium in Mean-Field Games (MFGs).
method Introduce P-MBED to measure model complexity, develop a novel exploration strategy, and establish polynomial sample complexity results.
result Learning Nash Equilibrium in MFGs is no more statistically challenging than solving a logarithmic number of single-agent RL problems.
New algorithms achieve uniform-PAC guarantees for RL with bounded eluder dimension.
problem Achieving strong performance guarantees in reinforcement learning.
method Proposes algorithms for nonlinear bandits and model-based episodic RL with a bounded eluder dimension.
result Achieves uniform-PAC sample complexity that matches state-of-the-art regret bounds or sample complexity guarantees.
We consider the problem of learning to optimize an unknown Markov decision process (MDP). We show that, if the MDP can be parameterized within some known function class, we can obtain regret bounds that scale with the dimensionality, rather than cardinality, of the system. We characterize this dependence explicitly as …
Develops a new model for RLHF accounting for partially observed states and intermediate feedback.
problem Lack of models for partially observed states and intermediate feedback in RLHF.
method PORRL model with cardinal and dueling feedback methods.
result Demonstrates improved learning and alignment with new model-based and model-free methods.
New insights on eluder dimension for function approximation in machine learning.
problem Complexity measure for online bandits and reinforcement learning with function approximation.
method Study the relationship between eluder dimension and generalized rank for different activation functions.
result Eluder dimension can be exponentially smaller or larger than generalized rank depending on the activation function.
Eluder dimension and information gain are equivalent for reproducing kernel Hilbert spaces.
problem Complexity measures in bandit and reinforcement learning.
method Equivalence of eluder dimension and information gain for reproducing kernel Hilbert spaces.
result Eluder dimension and information gain are equivalent for reproducing kernel Hilbert spaces.
Unified framework for efficient RL in MDP, POMDP, and PSR.
problem Sample efficient reinforcement learning under various decision-making frameworks.
method Proposed GEC complexity measure and a posterior sampling algorithm.
result Sample efficient learning achieved with sublinear regret bound in terms of GEC.
Study optimal policy regret in partially observable Markov games with adaptive opponents.
problem Optimal sequential decision-making in partially observable environments against strategic, adaptive opponents.
method An epoch-based optimistic maximum-likelihood algorithm that selects one policy per epoch using confidence sets built cumulatively from past data.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) policy regret for fixed problem parameters, with explicit dependence on horizon, adversary memory, confidence radius, and aggregate Eluder dimension. The paper develops algorithms for competitive RL in partially observable MGs.
problem Challenges in reinforcement learning with function approximation and partial observability.
method Proposes posterior sampling methods for self-play and adversarial learning in zero-sum MGs.
result Developed algorithms achieve low regret bounds scaling sublinearly with GEC and episode number.
New BE dimension measure reveals rich RL problems with sample-efficient algorithms.
problem Finding sample-efficient algorithms for complex RL problems.
method Introducing Bellman Eluder (BE) dimension and designing GOLF and OLIVE algorithms.
result GOLF and OLIVE algorithms learn near-optimal policies for low BE dimension problems with polynomial samples.
Paper proposes an algorithm for lifelong learning with shared structure.
problem Lifelong learning with shared structure in an online setting.
method Proposes a simple algorithm using multi-task empirical risk minimization.
result Establishes a sample complexity bound based on task-eluder dimension.
Paper develops efficient algorithms for zero-sum Markov games with general function classes.
problem Challenging settings in zero-sum Markov games with parameterized value functions or models.
method Developed new model-free and model-based algorithms for decoupled and coordinated settings.
result Improved sample complexity and regret bounds for various settings.
New RL algorithms achieve optimal policies with polynomial sample complexity for mean-field problems.
problem Statistical efficiency of Mean-Field Reinforcement Learning with general function approximation.
method Introduce MF-MBED to characterize problem complexity, propose algorithms based on maximal likelihood estimation.
result Rich mean-field RL problems have low MF-MBED, leading to polynomial sample complexity.
New findings on representation learning beyond linear functions.
problem Achieving diversity in representation learning beyond linear prediction functions.
method Analysis of eluder dimension and empirical risks.
result Diversity holds even with nonlinear prediction functions and multiple layers in neural networks.
New RL method reduces sample complexity for large policy spaces.
problem Large-scale RL with unknown optimal policies and state/action spaces.
method Introduces eluder dimension for policy space, proving near-optimal sample complexity.
result Near-optimal sample complexity upper bound that depends linearly on eluder dimension.
New RL algorithm reduces regret in finite-horizon episodic tasks.
problem Minimizing regret in model-based reinforcement learning.
method Optimism principle applied to value-targeted regression.
result Regret bound of i l d e O ( d H 3 T ) ilde{\mathcal{O}}(d\sqrt{H^{3}T}) i l d e O ( d H 3 T ) for linear mixtures. Paper develops efficient RL algorithm for general value function approximation.
problem Lack of theory for RL with general value function approximation.
method Provable efficient RL algorithm using bounded eluder dimension.
result Achieves a regret bound of O ~ ( p o l y ( d H ) T ) \widetilde{O}(\mathrm{poly}(dH)\sqrt{T}) O ( poly ( d H ) T ) . A new framework reduces RL sample complexity for complex MDPs.
problem Handling large state and action spaces in reinforcement learning.
method Unified model-based and model-free RL framework with ABC class, novel estimation function, and functional eluder dimension.
result OPERA algorithm achieves sample-efficient regret bounds for various MDP models.
Thompson Sampling is a well established approach to bandit and reinforcement learning problems. However its use in continuum armed bandit problems has received relatively little attention. We provide the first bounds on the regret of Thompson Sampling for continuum armed bandits under weak conditions on the function cl…
Study experiment planning with function approximation in contextual bandit problems.
problem Designing effective data collection strategies in settings with limited rewards.
method Proposes two experiment planning strategies compatible with function approximation.
result Eluder planning and sampling procedure achieves optimality guarantees.
This work improves sample efficiency in neural function approximation for reinforcement learning.
problem Improving sample efficiency in reinforcement learning with neural function approximation.
method Study of function approximation with two-layer neural networks (ReLU and polynomial activations) under generative and realizability models.
result Significant improvement in sample complexity compared to linear methods.
New algorithm LOOP learns infinite-horizon AMDPs efficiently with function approximation.
problem Learning optimal policies in infinite-horizon AMDPs with function approximation.
method LOOP combines model-based and value-based methods with novel confidence sets and policy updating.
result LOOP achieves sublinear regret bound of i l d e O ( p o l y ( d , s p ( V ∗ ) ) T β ) ilde{\mathcal{O}}(\mathrm{poly}(d, \mathrm{sp}(V^*)) \sqrt{Tβ} ) i l d e O ( poly ( d , sp ( V ∗ )) T β ) . Du, Kakade, Wang, and Yang recently established intriguing lower bounds on sample complexity, which suggest that reinforcement learning with a misspecified representation is intractable. Another line of work, which centers around a statistic called the eluder dimension, establishes tractability of problems similar to t…
Framework reduces contextual bandit learning to offline regression with near-optimal regret.
problem Efficient learning with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that minimizes regret with near-optimal oracle calls.
result Near-optimal regret for contextual bandits with large action spaces and O ( l o g ( T ) ) O(log(T)) O ( l o g ( T )) offline oracle calls. ENIAC method optimizes and explores complex RL problems with non-linear policies.
problem Theoretical understanding of non-linear policies in RL with strategic exploration.
method ENIAC, an actor-critic method for non-linear function approximation.
result ENIAC finds near-optimal policies in polynomial exploration rounds under bounded eluder dimension.
OE2D framework reduces contextual bandits to offline regression for near-optimal regret.
problem Efficiently learning contextual bandits with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that reduces contextual bandits to offline regression.
result Near-optimal regret for contextual bandits with large action spaces and O ( log T ) O(\log T) O ( log T ) calls to an offline regression oracle. Study contextual bandits with stage-wise constraints, proving regret bounds and extending results.
problem Contextual bandits with stage-wise constraints in high probability and expectation settings.
method Upper-confidence bound algorithms for linear and non-linear reward/cost functions, extending to multiple constraints.
result Regret bounds for various settings, including non-linear reward/cost functions.
New algorithm tackles non-stationary reinforcement learning with general function approximation.
problem Understanding non-stationary MDPs with function approximation.
method Dynamic Bellman Eluder (DBE) dimension for complexity, sliding window mechanism, confidence set design.
result Upper bound on dynamic regret for proposed SW-OPEA algorithm.
Unified framework for constrained online decision-making.
problem Sequential decisions under stage-wise feasibility constraints.
method Upper counterfactual confidence bounds and generalized eluder dimension.
result Principled foundation for constrained sequential decision-making.
The paper proposes a parallelizable clustering method for multivariate data.
problem The standard model-based clustering method assumes the same number of clusters per margin, which is often unrealistic.
method Developed a finite mixture model per margin with different numbers of clusters, and used a game-inspired algorithm to cluster multivariate data.
result The proposed method shows good performance in various scenarios and real datasets.
CPPO learns policies from partial offline data in MDPs with structural assumptions.
problem Offline Reinforcement Learning with partial coverage assumption.
method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.
result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.
OMLE combines optimism and MLE for efficient sequential decision making.
problem Efficiently solving sequential decision making problems, especially in partially observable settings.
method Combines optimism for exploration and maximum likelihood estimation for model learning.
result OMLE learns near-optimal policies for a wide range of sequential decision making problems.
The paper tackles causal bandits with unknown SCMs and soft interventions, providing upper and lower bounds on regret.
problem Optimizing interventions in a causal system with unknown SCMs and soft interventions.
method Assumes unknown SCMs from a general class, allows infinite interventions, and provides upper and lower bounds on regret.
result General upper and lower bounds on cumulative achievable regret for various SCMs.
Decomposes ultrametric spaces into scaled simplices.
problem Understanding the structure of ultrametric spaces.
method Introducing metric resolutions and coarse disjoint union.
result Constructs universal spaces for asymptotic dimension 0.
ARL-GEN adapts to the smallest model class in nested families for RL with improved regret.
problem Model selection for Reinforcement Learning with nested model families.
method Adaptive Reinforcement Learning (ARL-GEN) with value targeted regression and model selection module.
result ARL-GEN achieves a matching regret to an oracle with knowledge of the true model class.
The increased availability of the multi-view data (data on the same samples from multiple sources) has led to strong interest in models based on low-rank matrix factorizations. These models represent each data view via shared and individual components, and have been successfully applied for exploratory dimension reduct…
We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization. We establish that when the true value function lies wit…
New approach handles stochastic and partially-observable environments using discrete autoencoders and Monte Carlo tree search.
problem Challenges in planning for stochastic and partially-observable environments.
method Uses discrete autoencoders and a stochastic variant of Monte Carlo tree search.
result Significantly outperforms MuZero on stochastic chess and scales to DeepMind Lab.
Contextual bandits study how reward variance affects regret bounds.
problem Investigating how small reward variance impacts regret bounds in contextual bandits.
method Analyzing two types of adversaries and function approximation complexities.
result Regret bounds are influenced by the eluder dimension and reward variance.
Survey of integrating planning and learning in model-based reinforcement learning.
problem Sequential decision making in AI, formalized as MDP optimization.
method Systematic coverage of dynamics model learning and planning-learning integration.
result Broad conceptual overview of model-based reinforcement learning.
MoMA improves model-based RL by using unrestricted policy classes.
problem Limited sample efficiency and generalizability in model-based offline RL.
method Model-based mirror ascent algorithm with general function approximations.
result Theoretical guarantees and practical implementation of MoMA.
The effectiveness of model-based versus model-free methods is a long-standing question in reinforcement learning (RL). Motivated by recent empirical success of RL on continuous control tasks, we study the sample complexity of popular model-based and model-free algorithms on the Linear Quadratic Regulator (LQR). We show…
Algorithm adapts to shifting domains with minimal label queries.
problem Adaptive learning in online machine learning systems with domain shifts.
method Adaptive algorithm balancing regret and label queries for hidden domains.
result Achieves lower regret compared to uniform and greedy queries.
This paper analyzes how randomizing rewards in MBRL can improve performance without being overly optimistic.
problem The gap between theoretical worst-case regret analysis and empirical performance in MBRL.
method Reward randomization in model-based reinforcement learning (MBRL) with kernelized linear regulator (KNR) model.
result Reward randomization guarantees partial optimism and near-optimal worst-case regret.
Unified theory for kernel regression generalizes well under realistic assumptions.
problem Analyzing kernel regression under realistic conditions.
method Unified theory providing rigorous bounds for various settings.
result Self-regularization phenomenon in kernel matrices enables good generalization.
The use of kinetic modelling based on partial differential equations for the dynamics of stock price formation in financial markets is briefly reviewed. The importance of behavioral aspects in market booms and crashes and the role of agents' heterogeneity in emerging power laws for price distributions is emphasized and…
VO Q Q Q L optimizes RL with sparse rewards using weighted bounds.
problem Sparse rewards and non-linear function approximation in RL.
method VO Q Q Q L combines Q Q Q -learning with weighted bounds for optimal regret. result Achieves asymptotically optimal regret for linear function approximation.
A new dimension reduction method based on Gaussian finite mixtures is proposed as an extension to sliced inverse regression (SIR). The model-based SIR (MSIR) approach allows the main limitation of SIR to be overcome, i.e., failure in the presence of regression symmetric relationships, without the need to impose further…