arXiv research

The paper analyzes the sample complexities for policy evaluation with linear function approximation.

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

This work analyzes how neural networks learn representations in actor-critic algorithms.

Paper analyzes Greedy-GQ for reinforcement learning with Markovian noise.

This paper analyzes the sample complexity of two timescale reinforcement learning algorithms.

Paper analyzes normal approximation for two-timescale stochastic algorithms, revealing interaction between fast and slow timescales.

This study improves convergence of two-timescale SA under Markovian noise in reinforcement learning.

Introduces TDRC to balance TD's ease and soundness.

Two single-timescale algorithms improve TD learning with nonlinear approximations.

Study best-response learning dynamics in zero-sum polymatrix games under full and minimal information settings.

Study on distributional TD learning with linear approximations for better return estimation.

Linear two-timescale stochastic approximation (SA) scheme is an important class of algorithms which has become popular in reinforcement learning (RL), particularly for the policy evaluation problem. Recently, a number of works have been devoted to establishing the finite time analysis of the scheme, especially under th…

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Differ…

New algorithms improve distributional TD learning with linear approximations.

PFedRL-Rep learns shared and personalized policies for heterogeneous environments.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

In this paper we present a framework to analyze the asymptotic behavior of two timescale stochastic approximation algorithms including those with set-valued mean fields. This paper builds on the works of Borkar and Perkins & Leslie. The framework presented herein is more general as compared to the synchronous two times…

Improved TD learning with tail averaging and regularization achieves optimal convergence rates.

In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD($λ$) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) lear…

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…

Paper improves TD(0) convergence rate with LFA, i.i.d. samples, and averaging.

Proves convergence of neural networks in a two-timescale regime.

Uniform TD(0) bound derived for function approximation with Markov noise.

Paper analyzes TD($λ$) convergence rates for arbitrary features.

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their conver…

Paper shows TD learning without projection converges robustly.

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…

While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known. Here we take a first step towards extending theoretical convergence guarantees to TD learning with nonlinear…

Off-policy reinforcement learning has many applications including: learning from demonstration, learning multiple goal seeking policies in parallel, and representing predictive knowledge. Recently there has been an proliferation of new policy-evaluation algorithms that fill a longstanding algorithmic void in reinforcem…

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…

Learning the value function of a given policy (target policy) from the data samples obtained from a different policy (behavior policy) is an important problem in Reinforcement Learning (RL). This problem is studied under the setting of off-policy prediction. Temporal Difference (TD) learning algorithms are a popular cl…

Paper improves TD learning algorithm bounds with linear approx.

We provide non-asymptotic bounds for the well-known temporal difference learning algorithm TD(0) with linear function approximators. These include high-probability bounds as well as bounds in expectation. Our analysis suggests that a step-size inversely proportional to the number of iterations cannot guarantee optimal …

Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is one of the most widely used algorithms in reinforcement learning, its theoretical analysis has proved challenging and few guarantees on its s…

Temporal difference (TD) learning is a popular algorithm for policy evaluation in reinforcement learning, but the vanilla TD can substantially suffer from the inherent optimization variance. A variance reduced TD (VRTD) algorithm was proposed by Korda and La (2015), which applies the variance reduction technique direct…

TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates

This paper formalizes $Q$-learning and linear TD convergence using Lean 4.

Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.

New bounds show linear predictors rarely overfit with certain optimization methods.

Novel bounds improve TD learning consistency in RL.

Study on Adam-family methods for nonsmooth optimization with convergence guarantees.

Study analyzes a new algorithm for complex optimization problems.

Study optimal and instance-dependent guarantees for solving linear equations with Markovian data.

RO-TD learns sparse value functions efficiently.

Improved TD learning reduces batch sampling error.

Study shows TD(0) with linear approx. converges for reversible Markov chains.