arXiv research

New RL method reduces sample complexity for large state-action spaces.

New RL method handles large state-action spaces with complex models.

We present an efficient algorithm for model-free episodic reinforcement learning on large (potentially continuous) state-action spaces. Our algorithm is based on a novel $Q$-learning policy with adaptive data-driven discretization. The central idea is to maintain a finer partition of the state-action space in regions w…

ZoomRL learns efficient strategies for large state-action spaces using a metric.

Study optimal offline RL with uncertainty sets and distribution shifts.

Novel framework proves fast RL convergence in continuous spaces.

Adaptive discretization improves model-based RL in large spaces.

Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.

A new RL paradigm reduces state-action-value function approximation inefficiency.

GUM tackles MARL by avoiding overestimation through state-marginal restriction.

This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.

Develops a dynamic mean field theory for reinforcement learning.

Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model. We propose a parametric Q-learning algorithm that finds an approximate-optimal policy using a sample size proportional to the feature dimension $K$ and invariant …

SPAQL improves RL by adaptively partitioning state-action space and learning a time-invariant policy.

New method for natural policy gradients converges linearly.

When using reinforcement learning (RL) algorithms to evaluate a policy it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on the accuracy of the VF estimate, however, and determining…

We develop a normative framework for hierarchical model-based policy optimization based on applying second-order methods in the space of all possible state-action paths. The resulting natural path gradient performs policy updates in a manner which is sensitive to the long-range correlational structure of the induced st…

KeRNS tackles non-stationary reinforcement learning in metric spaces.

MO2 learns useful behaviours from past experience for new tasks.

Opportunistic spectrum access is one of the emerging techniques for maximizing throughput in congested bands and is enabled by predicting idle slots in spectrum. We propose a kernel-based reinforcement learning approach coupled with a novel budget-constrained sparsification technique that efficiently captures the envir…

We consider Markov Decision Processes (MDPs) where the rewards are unknown and may change in an adversarial manner. We provide an algorithm that achieves state-of-the-art regret bound of $O( \sqrt{τ(\ln|S|+\ln|A|)T}\ln(T))$, where $S$ is the state space, $A$ is the action space, $τ$ is the mixing time of the MDP, and $…

New RL approach uses future state and action visitation measures for better exploration.

New method recovers diverse policies from expert data using state-action pair weighting.

Paper introduces SALE for better state-action learning in RL.

We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the tas…

Model-free Reinforcement Learning (RL) algorithms such as Q-learning [Watkins, Dayan 92] have been widely used in practice and can achieve human level performance in applications such as video games [Mnih et al. 15]. Recently, equipped with the idea of optimism in the face of uncertainty, Q-learning algorithms [Jin, Al…

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameter…

Efficient RL in large POMDPs with latent determinism and embeddings.

Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…

Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.

Paper tackles covariate shift in offline IL using less proficient behavior data.

New framework for RL transfer learning with state-action mismatch.

New batch RL method avoids overly optimistic policies.

Develops a new reinforcement learning framework for complex control problems.

Most real-world problems have huge state and/or action spaces. Therefore, a naive application of existing tabular solution methods is not tractable on such problems. Nonetheless, these solution methods are quite useful if an agent has access to a relatively small state-action space homomorphism of the true environment …

Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.

Quantum algorithms speed up reinforcement learning policies in large state-action spaces.

A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.

A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…

New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.

This paper introduces CENIE to quantify environment novelty for better UED.

On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central …

A new method estimates multi-dimensional value distributions using Hilbert space embeddings.

RPO uses past and future state-action info for better policy optimization.

The paper improves Q-learning by incorporating pessimism for better sample efficiency.

Paper proposes a new RL approach combining IL and RL methods to improve decision-making.

Leveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity betwee…

Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games, and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data efficient Deep-Q-learning method…