arXiv research

Improved TD learning reduces batch sampling error.

Improved TD learning reduces variance and bias errors.

The study proposes using TD error for selecting σ in Q(σ, λ).

In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.

We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, w…

Unified framework for finite-sample RL algorithms using Lyapunov theory.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

New TD method stabilizes average-reward learning.

Paper improves TD(0) convergence rate with LFA, i.i.d. samples, and averaging.

TD learning reduces prediction error in Markov chain problems.

The paper identifies overfitting as the main bottleneck in efficient deep reinforcement learning.

TD-Flow improves long-term predictions in agent learning.

Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …

Study non-parametric value function estimation from a single path.

TDS provides exact samples for conditional distributions in diffusion models.

New bounds show linear predictors rarely overfit with certain optimization methods.

Study cyber-attacks on RL algorithms, focusing on cost signal manipulation.

Temporal Difference learning or TD($λ$) is a fundamental algorithm in the field of reinforcement learning. However, setting TD's $λ$ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the $λ$ selection problem as a bias-variance trade-off where the solution is …

Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

Paper improves TD learning algorithm bounds with linear approx.

This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source …

Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.

This paper presents a new neural architecture that combines a modulated Hebbian network (MOHN) with DQN, which we call modulated Hebbian plus Q network architecture (MOHQA). The hypothesis is that such a combination allows MOHQA to solve difficult partially observable Markov decision process (POMDP) problems which impa…

In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…

RO-TD learns sparse value functions efficiently.

We solve POMDPs by approximating them as finite-state MDPs.

Unified framework for PE and TD methods in continuous time and space.

Introduces TDRC to balance TD's ease and soundness.

A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…

Study shows TD(0) with linear approx. converges for reversible Markov chains.

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the $κ$-operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We i…

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their conver…

Uniform TD(0) bound derived for function approximation with Markov noise.

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

Improved TD(0) algorithm for reinforcement learning with linear approximations.

In reinforcement learning, the TD($λ$) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD($λ$) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…

Motivated by the widespread use of temporal-difference (TD-) and Q-learning algorithms in reinforcement learning, this paper studies a class of biased stochastic approximation (SA) procedures under a mild "ergodic-like" assumption on the underlying stochastic noise sequence. Building upon a carefully designed multistep…

Decentralized TD learning converges linearly with linear function approximation.

Paper analyzes SA for fixed-point equations with noise, establishing convergence rates.

Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…

New STDP rule for spiking neurons solves discrete action reinforcement learning tasks.

In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…

The paper analyzes the sample complexities for policy evaluation with linear function approximation.

A new estimator combines bootstrapping and rollout methods in RL.