arXiv research

Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.

TD learning reduces prediction error in Markov chain problems.

TD-Flow improves long-term predictions in agent learning.

New methods improve temporal difference learning for policy evaluation in Markov decision processes.

New algorithm optimally evaluates policies with linear approximations.

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

Improved TD learning reduces batch sampling error.

In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …

Temporal-difference and Q-learning learn feature representations that converge to optimal ones.

ReaPER improves learning efficiency by prioritizing reliable experiences.

A new method for reinforcement learning scales errors without tuning.

Paper improves TD learning algorithm bounds with linear approx.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

We consider the off-policy evaluation problem in Markov decision processes with function approximation. We propose a generalization of the recently introduced \emph{emphatic temporal differences} (ETD) algorithm \citep{SuttonMW15}, which encompasses the original ETD($λ$), as well as several other off-policy evaluation …

Paper introduces a new reinforcement learning method with improved performance.

Proposes a method to enhance exploration in RL using temporal difference uncertainties.

New loss function equivalence reveals PER's uniform sampling can be improved.

New TD method stabilizes average-reward learning.

Study on natural actor-critic for POMDPs with finite memory.

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.

Temporal difference (TD) learning is a popular algorithm for policy evaluation in reinforcement learning, but the vanilla TD can substantially suffer from the inherent optimization variance. A variance reduced TD (VRTD) algorithm was proposed by Korda and La (2015), which applies the variance reduction technique direct…

Unified framework for PE and TD methods in continuous time and space.

Study optimizes solving fixed-point equations using subspace search.

Kernel-based methods improve policy evaluation in MRP models.

We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, w…

The paper identifies overfitting as the main bottleneck in efficient deep reinforcement learning.

Temporal-Difference learning (TD) [Sutton, 1988] with function approximation can converge to solutions that are worse than those obtained by Monte-Carlo regression, even in the simple case of on-policy evaluation. To increase our understanding of the problem, we investigate the issue of approximation errors in areas of…

Temporal difference learning explained through gradient splitting, improving convergence times.

A novel approach uses an ensemble of Gaussian processes for robust and adaptive reinforcement learning.

Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …

Recently, a new multi-step temporal learning algorithm, called $Q(σ)$, unifies $n$-step Tree-Backup (when $σ=0$) and $n$-step Sarsa (when $σ=1$) by introducing a sampling parameter $σ$. However, similar to other multi-step temporal-difference learning algorithms, $Q(σ)$ needs much memory consumption and computation tim…

This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source …

Paper develops bounds for stochastic approximation with averaging.

Quantile TD learning outperforms classical TD learning for value estimation.

UVU simplifies value uncertainty quantification in RL.

Study on reinforcement learning dynamics using statistical physics.

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the $κ$-operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We i…

Emotions are intimately tied to motivation and the adaptation of behavior, and many animal species show evidence of emotions in their behavior. Therefore, emotions must be related to powerful mechanisms that aid survival, and, emotions must be evolutionary continuous phenomena. How and why did emotions evolve in nature…

Temporal Difference learning or TD($λ$) is a fundamental algorithm in the field of reinforcement learning. However, setting TD's $λ$ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the $λ$ selection problem as a bias-variance trade-off where the solution is …

We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distributional}. In the policy evaluation setting, we design two new algorithms called distributional GTD2 a…

Paper improves TD(0) convergence rate with LFA, i.i.d. samples, and averaging.

Here, we provide a supplementary material for Takayuki Osogami, "Uncorrected least-squares temporal difference with lambda-return," which appears in {\it Proceedings of the 34th AAAI Conference on Artificial Intelligence} (AAAI-20).

Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-S…

The paper analyzes how over-parameterization affects reinforcement learning performance.

Quantile Temporal-Difference learning proved convergent with proof.

Paper analyzes statistical efficiency of TD learning in Hilbert spaces with Freedman's inequality.

Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the …