arXiv research

Paper proposes an active multi-step TD algorithm for reinforcement learning.

Multi-step temporal difference (TD) learning is an important approach in reinforcement learning, as it unifies one-step TD learning with Monte Carlo methods in a way where intermediate algorithms can outperform either extreme. They address a bias-variance trade off between reliance on current estimates, which could be …

Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($σ$) algorithm (Sutton and Barto (2017)) unifies both. This paper extends the Q($σ$) algorithm to an online multi-step algorithm Q($σ, λ$) using eligibility traces and int…

Study non-parametric value function estimation from a single path.

New algorithms improve reinforcement learning stability and performance.

The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.

Decentralized TD learning converges linearly with linear function approximation.

MB-DQN uses different backup lengths for improved reinforcement learning.

We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficien…

Modern deep reinforcement learning methods have departed from the incremental learning required for eligibility traces, rendering the implementation of the $λ$-return difficult in this context. In particular, off-policy methods that utilize experience replay remain problematic because their random sampling of minibatch…

New method improves deep RL by combining emphatic weightings with replay data.

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…

RO-TD learns sparse value functions efficiently.

Introduces TDRC to balance TD's ease and soundness.

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their conver…

Improved TD learning reduces batch sampling error.

Uniform TD(0) bound derived for function approximation with Markov noise.

New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.

New TD method stabilizes average-reward learning.

Improved TD(0) algorithm for reinforcement learning with linear approximations.

In reinforcement learning, the TD($λ$) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD($λ$) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…

Improved TD learning reduces variance and bias errors.

Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…

The study proposes using TD error for selecting σ in Q(σ, λ).

Study on distributional TD learning with linear approximations for better return estimation.

While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known. Here we take a first step towards extending theoretical convergence guarantees to TD learning with nonlinear…

We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, w…

Unified framework for finite-sample RL algorithms using Lyapunov theory.

The problem of on-line off-policy evaluation (OPE) has been actively studied in the last decade due to its importance both as a stand-alone problem and as a module in a policy improvement scheme. However, most Temporal Difference (TD) based solutions ignore the discrepancy between the stationary distribution of the beh…

In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …

TD learning reduces interference, leading to better generalization.

The paper calculates prices for multi-step barrier options under the Black-Scholes model.

Accelerates TD learning for long-horizon reinforcement learning problems.

Temporal-difference (TD) networks are a class of predictive state representations that use well-established TD methods to learn models of partially observable dynamical systems. Previous research with TD networks has dealt only with dynamical systems with finite sets of observations and actions. We present an algorithm…

Efficiently optimizes expensive functions with multi-step lookahead using one-shot optimization.

Paper shows TD learning without projection converges robustly.

Maps from buildings to spaces study K-theory of Hecke algebras.

Paper improves TD(0) convergence rate with LFA, i.i.d. samples, and averaging.

TDS provides exact samples for conditional distributions in diffusion models.

Paper presents a copula-based method to efficiently generate correlated sample paths from multi-step time series models.

The paper explores dynamic ensembles for multi-step forecasting.

Proposes a convergent TD algorithm for off-policy RL.

QAM uses adjoint matching to optimize continuous-action RL policies efficiently.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

Paper analyzes TD($λ$) convergence rates for arbitrary features.