arXiv research

New method finds unseen states for RL, improving performance.

Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms---to estimate value functions---continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the prop…

Despite recent successes in Reinforcement Learning, value-based methods often suffer from high variance hindering performance. In this paper, we illustrate this in a continuous control setting where state of the art methods perform poorly whenever sensor noise is introduced. To overcome this issue, we introduce Recurre…

Study improves $L^{\infty}$ estimates and extreme value behavior in stochastic differential games.

ESAC improves reinforcement learning by lookahead and intuition.

UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.

Dyna is an architecture for model-based reinforcement learning (RL), where simulated experience from a model is used to update policies or value functions. A key component of Dyna is search-control, the mechanism to generate the state and action from which the agent queries the model, which remains largely unexplored. …

At the working heart of policy iteration algorithms commonly used and studied in the discounted setting of reinforcement learning, the policy evaluation step estimates the value of states with samples from a Markov reward process induced by following a Markov policy in a Markov decision process. We propose a simple and…

TD learning reduces prediction error in Markov chain problems.

Low-rank forecasting improves consistency in time series predictions.

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…

In this paper, we introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that…

This paper improves reinforcement learning policies in a scalable way.

QR-MIX models joint state-action values as a distribution to handle randomness in MARL.

Improved Shapley Values for tree-based models, more accurate than existing methods.

New method estimates optimal Q-values with better accuracy for specific problems.

New method improves deep policy gradient algorithms by learning relative state values.

A nonparametric approach for policy learning for POMDPs is proposed. The approach represents distributions over the states, observations, and actions as embeddings in feature spaces, which are reproducing kernel Hilbert spaces. Distributions over states given the observations are obtained by applying the kernel Bayes' …

A new method for estimating joint value functions in multi-scene reinforcement learning.

This work considers two distinct settings: imitation learning and goal-conditioned reinforcement learning. In either case, effective solutions require the agent to reliably reach a specified state (a goal), or set of states (a demonstration). Drawing a connection between probabilistic long-term dynamics and the desired…

This paper studies optimal approximation factors in misspecified off-policy RL, identifying key factors under various settings.

Model-free reinforcement learning methods such as the Proximal Policy Optimization algorithm (PPO) have successfully applied in complex decision-making problems such as Atari games. However, these methods suffer from high variances and high sample complexity. On the other hand, model-based reinforcement learning method…

OSIRIS reduces variance in off-policy evaluation by omitting irrelevant states.

Perry uses auxiliary data to estimate RL policy values with confidence intervals.

New methods estimate policy value and gradients for deterministic policies from off-policy data.

Quantum SVT reduces credit risk analysis costs.

A new imputation method MissARF uses adversarial random forests for fast and accurate missing value imputation.

Proposes a method to enhance exploration in RL using temporal difference uncertainties.

A novel approach uses an ensemble of Gaussian processes for robust and adaptive reinforcement learning.

A new method learns action representations for reinforcement learning.

We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…

The objective is to study an on-line Hidden Markov model (HMM) estimation-based Q-learning algorithm for partially observable Markov decision process (POMDP) on finite state and action sets. When the full state observation is available, Q-learning finds the optimal action-value function given the current action (Q func…

We consider the estimation of the policy gradient in partially observable Markov decision processes (POMDP) with a special class of structured policies that are finite-state controllers. We show that the gradient estimation can be done in the Actor-Critic framework, by making the critic compute a "value" function that …

Paper uses VAEAC to estimate Shapley values for complex models with mixed features.

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…

Paper improves reinforcement learning in multi-scene tasks.

In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optima…

We investigate the predictability of several range-based stock volatility estimators, and compare them to the standard close-to-close estimator which is most commonly acknowledged as the volatility. The patterns of volatility changes are analyzed using LSTM recurrent neural networks, which are a state of the art method…

We introduce the active exploration problem in Markov decision processes (MDPs). Each state of the MDP is characterized by a random value and the learner should gather samples to estimate the mean value of each state as accurately as possible. Similarly to active exploration in multi-armed bandit (MAB), states may have…

The paper tackles reward-relevance in offline RL with sparse decision dynamics.

A new metric detects non-Markovian states in partially observable environments.

We describe the Customer LifeTime Value (CLTV) prediction system deployed at ASOS.com, a global online fashion retailer. CLTV prediction is an important problem in e-commerce where an accurate estimate of future value allows retailers to effectively allocate marketing spend, identify and nurture high value customers an…

Develops hierarchical reinforcement learning value function approximators.

Many reinforcement learning algorithms use value functions to guide the search for better policies. These methods estimate the value of a single policy while generalizing across many states. The core idea of this paper is to flip this convention and estimate the value of many policies, for a single set of states. This …

Particle MCMC is a class of algorithms that can be used to analyse state-space models. They use MCMC moves to update the parameters of the models, and particle filters to propose values for the path of the state-space model. Currently the default is to use random walk Metropolis to update the parameter values. We show …

EASE estimator improves probabilistic value estimation efficiency.

Study on future-dependent value functions for off-policy evaluation in complex environments.

In this analytical study we derive the optimal unbiased value estimator (MVU) and compare its statistical risk to three well known value estimators: Temporal Difference learning (TD), Monte Carlo estimation (MC) and Least-Squares Temporal Difference Learning (LSTD). We demonstrate that LSTD is equivalent to the MVU if …