arXiv research

TD learning reduces prediction error in Markov chain problems.

Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.

TD-Flow improves long-term predictions in agent learning.

A new method for reinforcement learning scales errors without tuning.

New methods improve temporal difference learning for policy evaluation in Markov decision processes.

ReaPER improves learning efficiency by prioritizing reliable experiences.

Improves spatio-temporal forecasting by reducing errors between training and inference.

New algorithm optimally evaluates policies with linear approximations.

Improved TD learning reduces batch sampling error.

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …

Paper improves TD learning algorithm bounds with linear approx.

Recent results in coupled or temporal graphical models offer schemes for estimating the relationship structure between features when the data come from related (but distinct) longitudinal sources. A novel application of these ideas is for analyzing group-level differences, i.e., in identifying if trends of estimated ob…

Temporal-difference and Q-learning learn feature representations that converge to optimal ones.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

We consider the off-policy evaluation problem in Markov decision processes with function approximation. We propose a generalization of the recently introduced \emph{emphatic temporal differences} (ETD) algorithm \citep{SuttonMW15}, which encompasses the original ETD($λ$), as well as several other off-policy evaluation …

Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the …

Proposes a method to enhance exploration in RL using temporal difference uncertainties.

Flow prediction (e.g., crowd flow, traffic flow) with features of spatial-temporal is increasingly investigated in AI research field. It is very challenging due to the complicated spatial dependencies between different locations and dynamic temporal dependencies among different time intervals. Although measurements of …

Paper tackles temporal overfitting in wind power curve modeling.

New loss function equivalence reveals PER's uniform sampling can be improved.

Paper introduces a new reinforcement learning method with improved performance.

Study on natural actor-critic for POMDPs with finite memory.

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

New insights into RL efficiency from managing time discretization.

New TD method stabilizes average-reward learning.

Unified framework for PE and TD methods in continuous time and space.

Classifiers are among the most widely used supervised machine learning algorithms. Many classification models exist, and choosing the right one for a given task is difficult. During model selection and debugging, data scientists need to assess classifiers' performances, evaluate their learning behavior over time, and c…

Temporal difference (TD) learning is a popular algorithm for policy evaluation in reinforcement learning, but the vanilla TD can substantially suffer from the inherent optimization variance. A variance reduced TD (VRTD) algorithm was proposed by Korda and La (2015), which applies the variance reduction technique direct…

Study non-parametric value function estimation from a single path.

Proposes a non-autoregressive Transformer for time series forecasting.

Study optimizes solving fixed-point equations using subspace search.

Develops models for temporally abstract reasoning and attention.

The paper identifies overfitting as the main bottleneck in efficient deep reinforcement learning.

Recently, a new multi-step temporal learning algorithm, called $Q(σ)$, unifies $n$-step Tree-Backup (when $σ=0$) and $n$-step Sarsa (when $σ=1$) by introducing a sampling parameter $σ$. However, similar to other multi-step temporal-difference learning algorithms, $Q(σ)$ needs much memory consumption and computation tim…

New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.

Kernel-based methods improve policy evaluation in MRP models.

We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, w…

Estimates mean of distributed vectors with sparsification and spatial/temporal correlations.

FPG uses fractional calculus for efficient reinforcement learning with long-term memory.

Temporal-Difference learning (TD) [Sutton, 1988] with function approximation can converge to solutions that are worse than those obtained by Monte-Carlo regression, even in the simple case of on-policy evaluation. To increase our understanding of the problem, we investigate the issue of approximation errors in areas of…

A novel approach uses an ensemble of Gaussian processes for robust and adaptive reinforcement learning.

Meta-learning adjusts TD learning's eligibility trace parameter for more efficient reinforcement learning.

Study on reinforcement learning dynamics using statistical physics.

Different optimizer choices lead to different financial model predictions.

Agents compose pre-trained policies for complex tasks, improving zero-shot performance.

Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …

This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source …