arXiv research

Deep RBVFs improve continuous control in RL.

In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low d…

Develops hierarchical reinforcement learning value function approximators.

We study the use of randomized value functions to guide deep exploration in reinforcement learning. This offers an elegant means for synthesizing statistically and computationally efficient exploration with common practical approaches to value function learning. We present several reinforcement learning algorithms that…

New method calculates Shapley values for uncertain functions.

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation t…

PBVFs generalize across policies using learned value functions.

Boosts Q-learning by using value function bounds.

New RL method learns value function for many policies using few key states.

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local traj…

Paper presents an efficient exploration method for reinforcement learning.

Novel framework for Bayesian reinforcement learning infers value function distributions.

New method for off-policy evaluation in POMDPs using future-dependent value functions.

This paper studies a recent proposal to use randomized value functions to drive exploration in reinforcement learning. These randomized value functions are generated by injecting random noise into the training data, making the approach compatible with many popular methods for estimating parameterized value functions. B…

Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms---to estimate value functions---continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the prop…

A reinforcement learning framework combining value function and tree search planner for strategic and tactical decisions.

In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…

Paper develops efficient RL algorithm for general value function approximation.

We analyze the Gambler's problem, finding a complex, fractal value function.

Despite recent successes in Reinforcement Learning, value-based methods often suffer from high variance hindering performance. In this paper, we illustrate this in a continuous control setting where state of the art methods perform poorly whenever sensor noise is introduced. To overcome this issue, we introduce Recurre…

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structu…

A novel approach uses an ensemble of Gaussian processes for robust and adaptive reinforcement learning.

PPG separates policy and value function training phases for better reinforcement learning efficiency.

This paper introduces a new framework for learning nearly decomposable Q-functions via communication minimization.

A new method for estimating joint value functions in multi-scene reinforcement learning.

Paper introduces a new value function for state transitions and optimal policy learning.

This paper interpolates reward functions to predict optimal value functions in MORL.

In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with th…

Time-variant value function transfer method for RL.

A new RL paradigm reduces state-action-value function approximation inefficiency.

We propose a flexible method for estimating value functions in reinforcement learning without parametric assumptions.

UVU simplifies value uncertainty quantification in RL.

Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservati…

ZOSPI improves RL policies with global value function exploitation.

The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. …

We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization. We establish that when the true value function lies wit…

We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of least-squares value iteration that use Boltzmann or epsilon-greedy exploration can be highly ineffic…

Offline RL struggles with sample efficiency due to fundamental barriers.

Improved TD learning reduces batch sampling error.

Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function $V$. Two novel algorithms are proposed to approximate the true value…

Paper tackles transfer RL under unobserved context, developing methods to reduce bias.

We present a framework to derive risk bounds for vector-valued learning with a broad class of feature maps and loss functions. Multi-task learning and one-vs-all multi-category learning are treated as examples. We discuss in detail vector-valued functions with one hidden layer, and demonstrate that the conditions under…

RO-TD learns sparse value functions efficiently.

This paper shows using classification instead of regression improves deep RL scalability.

The logcosh loss function helps neural networks learn set-valued functions better.

A new estimator for evaluating policies in unknown environments.

Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.

Kernel smoothing improves LLM reasoning efficiency.