arXiv research

Study on distributional TD learning with linear approximations for better return estimation.

Accelerates TD learning for long-horizon reinforcement learning problems.

RO-TD learns sparse value functions efficiently.

New algorithms improve distributional TD learning with linear approximations.

While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known. Here we take a first step towards extending theoretical convergence guarantees to TD learning with nonlinear…

Improved TD learning with neural nets reduces sample complexity and overparameterization.

Quantile TD learning outperforms classical TD learning for value estimation.

Uniform TD(0) bound derived for function approximation with Markov noise.

Novel bounds improve TD learning consistency in RL.

The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.

Value functions derived from Markov decision processes arise as a central component of algorithms as well as performance metrics in many statistics and engineering applications of machine learning techniques. Computation of the solution to the associated Bellman equations is challenging in most practical cases of inter…

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their conver…

Meta-learning adjusts TD learning's eligibility trace parameter for more efficient reinforcement learning.

Paper shows TD learning without projection converges robustly.

We solve POMDPs by approximating them as finite-state MDPs.

In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…

Stochastic differential equation approximation for linear TD(0) under Markovian noise

New method makes reinforcement learning robust to heavy-tailed rewards.

Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learnin…

Two single-timescale algorithms improve TD learning with nonlinear approximations.

Multi-step temporal difference (TD) learning is an important approach in reinforcement learning, as it unifies one-step TD learning with Monte Carlo methods in a way where intermediate algorithms can outperform either extreme. They address a bias-variance trade off between reliance on current estimates, which could be …

Stochastic Approximation (SA) is a popular approach for solving fixed-point equations where the information is corrupted by noise. In this paper, we consider an SA involving a contraction mapping with respect to an arbitrary norm, and show its finite-sample error bounds while using different stepsizes. The idea is to c…

Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

TD learning reduces interference, leading to better generalization.

Supervised learning alone can be effective for offline RL, revealing essential elements.

New statistical methods improve TD learning for policy evaluation.

In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with th…

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

Study best-response learning dynamics in zero-sum polymatrix games under full and minimal information settings.

Temporal difference (TD) learning is a popular algorithm for policy evaluation in reinforcement learning, but the vanilla TD can substantially suffer from the inherent optimization variance. A variance reduced TD (VRTD) algorithm was proposed by Korda and La (2015), which applies the variance reduction technique direct…

We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We first consider this problem under a certain scaling of the approximating function, leading to a regime called lazy tr…

There is a long history of using meta learning as representation learning, specifically for determining the relevance of inputs. In this paper, we examine an instance of meta-learning in which feature relevance is learned by adapting step size parameters of stochastic gradient descent---building on a variety of prior w…

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

Introduces TDRC to balance TD's ease and soundness.

Optimized portfolio management with dynamic market regimes using RL and OC learning.

The paper analyzes stability of random matrix products with Markovian noise.

Motivated by the emerging use of multi-agent reinforcement learning (MARL) in engineering applications such as networked robotics, swarming drones, and sensor networks, we investigate the policy evaluation problem in a fully decentralized setting, using temporal-difference (TD) learning with linear function approximati…

We seek to learn an effective policy for a Markov Decision Process (MDP) with continuous states via Q-Learning. Given a set of basis functions over state action pairs we search for a corresponding set of linear weights that minimizes the mean Bellman residual. Our algorithm uses a Kalman filter model to estimate those …

Improved TD learning reduces batch sampling error.

New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.

New method uses MCTS only at test time for faster game learning.

Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($σ$) algorithm (Sutton and Barto (2017)) unifies both. This paper extends the Q($σ$) algorithm to an online multi-step algorithm Q($σ, λ$) using eligibility traces and int…

Paper improves confidence intervals for LSA with multiplier bootstrap.

Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function $V$. Two novel algorithms are proposed to approximate the true value…

Paper improves TD learning algorithm bounds with linear approx.

A new correction term improves sample efficiency in deep reinforcement learning.

Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is one of the most widely used algorithms in reinforcement learning, its theoretical analysis has proved challenging and few guarantees on its s…