arXiv research

Approximate models help RL by reducing policy search space.

Near-optimal per-action regret bounds for sleeping bandits are derived.

New algorithm reduces regret by focusing on near-optimal actions.

Solves action selection for large spaces in RL, achieving near-optimal performance.

Most real-world problems have huge state and/or action spaces. Therefore, a naive application of existing tabular solution methods is not tractable on such problems. Nonetheless, these solution methods are quite useful if an agent has access to a relatively small state-action space homomorphism of the true environment …

New method achieves near-optimal regret without simulator.

We present the first PAC optimal algorithm for Bayes-Adaptive Markov Decision Processes (BAMDPs) in continuous state and action spaces, to the best of our knowledge. The BAMDP framework elegantly addresses model uncertainty by incorporating Bayesian belief updates into long-term expected return. However, computing an e…

Any reinforcement learning algorithm that applies to all Markov decision processes (MDPs) will suffer $Ω(\sqrt{SAT})$ regret on some MDP, where $T$ is the elapsed time and $S$ and $A$ are the cardinalities of the state and action spaces. This implies $T = Ω(SA)$ time to guarantee a near-optimal policy. In many settings…

Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this infer…

The paper extends Thompson Sampling to infinite action spaces using information theory.

Algorithm reduces regret in safe Bayesian optimization with monotonicity constraints.

New algorithm finds near-optimal policies efficiently in zero-sum games.

We provide a simple and efficient algorithm for adversarial $k$-action $d$-outcome non-degenerate locally observable partial monitoring game for which the $n$-round minimax regret is bounded by $6(d+1) k^{3/2} \sqrt{n \log(k)}$, matching the best known information-theoretic upper bound. The same algorithm also achieves…

In this paper, we settle the sampling complexity of solving discounted two-player turn-based zero-sum stochastic games up to polylogarithmic factors. Given a stochastic game with discount factor $γ\in(0,1)$ we provide an algorithm that computes an $ε$-optimal strategy with high-probability given $\tilde{O}((1 - γ)^{-3}…

Study lenient regret and good-action identification in Gaussian process bandits.

Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions. Unfortunately, existing approaches t…

A new RL approach learns near-equivalent actions for healthcare decisions.

Framework reduces contextual bandit learning to offline regression with near-optimal regret.

Information-theoretic Bayesian regret bounds of Russo and Van Roy capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among o…

We consider interactive learning and covering problems, in a setting where actions may incur different costs, depending on the response to the action. We propose a natural greedy algorithm for response-dependent costs. We bound the approximation factor of this greedy algorithm in active learning settings as well as in …

OE2D framework reduces contextual bandits to offline regression for near-optimal regret.

Study shows reinforcement learning algorithm's performance depends on metric space's size.

Develops a new RL algorithm for medical treatment regimes.

Stochastic Q-learning tackles large action spaces with reduced computation.

A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.

New algorithm handles both decaying and non-decaying bandit problems.

We study an exploration method for model-free RL that generalizes the counter-based exploration bonus methods and takes into account long term exploratory value of actions rather than a single step look-ahead. We propose a model-free RL method that modifies Delayed Q-learning and utilizes the long-term exploration bonu…

Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy …

New algorithm reduces sample complexity for planning in MDPs.

We study Imitation Learning (IL) from Observations alone (ILFO) in large-scale MDPs. While most IL algorithms rely on an expert to directly provide actions to the learner, in this setting the expert only supplies sequences of observations. We design a new model-free algorithm for ILFO, Forward Adversarial Imitation Lea…

StepMix algorithm ensures safe exploration in reinforcement learning with near-optimal performance.

Paper studies S-rectangular DR-RL models for robust reinforcement learning with near-optimal sample complexity.

New method reduces ensemble size for linear bandits, achieving near optimal regret.

New algorithm tackles nonstationary linear bandits with latent dynamics.

The use of Reinforcement Learning (RL) is still restricted to simulation or to enhance human-operated systems through recommendations. Real-world environments (e.g. industrial robots or power grids) are generally designed with safety constraints in mind implemented in the shape of valid actions masks or contingency con…

The paper proposes a principle for dynamically adjusting the granularity of reinforcement learning abstractions.

Neural Index Policy for multi-action bandits with heterogeneous budgets.

In the real world, agents often have to operate in situations with incomplete information, limited sensing capabilities, and inherently stochastic environments, making individual observations incomplete and unreliable. Moreover, in many situations it is preferable to delay a decision rather than run the risk of making …

We study the problem of multi-agent reinforcement learning (MARL) with adaptivity constraints -- a new problem motivated by real-world applications where deployments of new policies are costly and the number of policy updates must be minimized. For two-player zero-sum Markov Games, we design a (policy) elimination base…

Paper solves learning imperfect-information games with fewer episodes.

We address the problem of regret minimization in logistic contextual bandits, where a learner decides among sequential actions or arms given their respective contexts to maximize binary rewards. Using a fast inference procedure with Polya-Gamma distributed augmentation variables, we propose an improved version of Thomp…

Bayesian method estimates dynamics from near-optimal trajectories.

Paper tackles non-stationary kernelized bandits with near-optimal algorithm.

Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.

We initiate the study of multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system extending recent results for the special case of stochastic bandits. We provide a framework which modifies the aggressive exploration enjoyed b…

Hybrid Policy Optimization tackles reinforcement learning in hybrid spaces, improving performance over PPO.

Study of reinforcement learning with additional feedback observations.

New RL algorithm ensures stable, replicable policies.