Improved TD learning for non-i.i.d. Markovian data.
problem Convergence analysis of two time-scale TD learning under Markovian samples.
method Non-asymptotic convergence analysis of two time-scale TD with gradient correction under Markovian data.
result Two time-scale TD can converge as fast as O(log t/(t^(2/3))) under diminishing stepsize.
Q(Δ)-Learning improves Q-Learning by separating action-value functions into different time scales.
problem Q-Learning struggles with bias-variance trade-off, especially in long-term rewards.
method Introduces Q(Δ)-Learning, extending TD(Δ) to decompose Q(Δ)-function into distinct discount factors. result Q(Δ)-Learning achieves better stability and scalability, especially for long-term tasks. In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with th…
RO-TD learns sparse value functions efficiently.
problem Learning sparse value functions efficiently.
method RO-TD integrates off-policy convergent gradient TD methods and online convex regularization.
result RO-TD learns sparse value functions with low computational complexity.
The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their conver…
Paper proposes an active multi-step TD algorithm for reinforcement learning.
problem Challenging decision making and control tasks in reinforcement learning.
method Active stepsize learning and adaptive multi-step TD algorithm with context-aware mechanism.
result Competitive results compared to other reinforcement learning baselines on discrete and continuous space tasks.
We establish decoupled functional CLTs for two-time-scale stochastic approximation.
problem Understanding the asymptotic behavior of two-time-scale stochastic approximation.
method Martingale problem approach and auxiliary sequence.
result The limiting dynamics of two-time-scale SA are independent of each other.
We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, w…
In unsupervised domain adaptation (UDA), classifiers for the target domain (TD) are trained with clean labeled data from the source domain (SD) and unlabeled data from TD. However, in the wild, it is difficult to acquire a large amount of perfectly clean labeled data in SD given limited budget. Hence, we consider a new…
TD learning reduces interference, leading to better generalization.
problem Understanding and reducing interference in TD learning for better generalization.
method Analyzing the inner product of gradients as interference, comparing TD and supervised learning, and examining the dynamics of interference and bootstrapping.
result TD learning leads to low-interference, under-generalizing parameters, while supervised learning does the opposite.
The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…
Off-policy reinforcement learning has many applications including: learning from demonstration, learning multiple goal seeking policies in parallel, and representing predictive knowledge. Recently there has been an proliferation of new policy-evaluation algorithms that fill a longstanding algorithmic void in reinforcem…
We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.
problem Achieving decoupled convergence in nonlinear two-time-scale stochastic approximation.
method Nested local linearity assumption, suitable step size selection, convergence analysis of matrix cross term, fourth-order moment convergence rates.
result Finite-time decoupled convergence rates can be achieved in nonlinear two-time-scale stochastic approximation with proper step size selection.
Introduces TDRC to balance TD's ease and soundness.
problem TD learning's instability and divergence issues.
method Gradient Temporal-Difference Learning with Regularized Corrections (TDRC).
result TDRC performs as well as TD when TD works, but is sound in divergent cases.
Improved TD learning reduces batch sampling error.
problem TD learning can be inaccurate in batch settings due to action weighting.
method Introduced PSEC-TD(0) to correct action weighting using importance sampling.
result PSEC-TD(0) produces more accurate value function estimates.
Uniform TD(0) bound derived for function approximation with Markov noise.
problem Uniform concentration bound for TD(0) with function approximation.
method Contractive stochastic approximation, martingale and Markov noises, Poisson equation, relaxed concentration inequalities.
result Uniform all-time concentration bound for TD(0) with linear function approximation.
TDS provides exact samples for conditional distributions in diffusion models.
problem Lack of exact sampling methods for diffusion models.
method Sequential Monte Carlo (SMC) algorithm with twisting technique.
result TDS offers more accurate approximations with fewer particles compared to heuristics.
We provide non-asymptotic bounds for the well-known temporal difference learning algorithm TD(0) with linear function approximators. These include high-probability bounds as well as bounds in expectation. Our analysis suggests that a step-size inversely proportional to the number of iterations cannot guarantee optimal …
New analysis of stochastic approximation with non-expansive mappings.
problem Finite-time analysis of two-time-scale stochastic approximation with non-expansive mappings.
method Studied two-time-scale stochastic approximation algorithms with non-expansive mappings and projection steps.
result Last-iterate mean square residual error decays at a rate O(1/k1/4−ε). In this paper we revisit the method of off-policy corrections for reinforcement learning (COP-TD) pioneered by Hallak et al. (2017). Under this method, online updates to the value function are reweighted to avoid divergence issues typical of off-policy learning. While Hallak et al.'s solution is appealing, it cannot ea…
New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
problem TD learning's sensitivity to step size specification.
method Implicit TD algorithms reformulate TD updates into fixed point equations.
result Implicit TD algorithms are more stable and less sensitive to step size.
Improved bounds for non-linear SA with fast convergence.
problem Stochastic approximation with non-linear mappings and multiple time scales.
method Mean squared error bounds with O(1/k) rate for contractive mappings. result First O(1/k) rate for non-linear two-time-scale SA without additional smoothness assumptions. Improved TD learning with neural nets reduces sample complexity and overparameterization.
problem Temporal difference learning with neural networks in large state spaces.
method Projection-free and max-norm regularized Neural TD learning, with Lyapunov drift analysis.
result Max-norm regularization significantly improves TD learning's sample complexity and overparameterization.
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD(λ). result Improved numerical stability and broader step-size range.
Paper analyzes convergence rates of two time-scale AC and NAC algorithms.
problem Finite-sample convergence rate analysis of two time-scale AC and NAC algorithms.
method Developed novel techniques for bias error and convergence rate analysis.
result Established non-asymptotic convergence rates for two time-scale AC and NAC.
Improved TD(0) algorithm for reinforcement learning with linear approximations.
problem Sensitivity of TD(0) to stepsizes and slow convergence.
method AdaTD(0) - Adaptive projected variant of TD(0) with linear function approximation.
result AdaTD(0) converges faster and is less sensitive to stepsizes.
In reinforcement learning, the TD(λ) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD(λ) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…
TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates
problem TD(0) learning under Markovian sampling
method Polyak-Ruppert averaging with a single stepsize
result Simultaneous high-probability convergence guarantees for TD(0) iterates and PR average
Proposes Topology Distance for evaluating GANs.
problem Challenges in evaluating GANs' goodness.
method Builds Vietoris-Rips complex on image features and defines TD based on latent manifold comparisons.
result Demonstrates TD's superiority over existing metrics.
Improved TD learning reduces variance and bias errors.
problem Inefficient optimization variance in TD learning.
method Proposed a mathematically solid analysis of VRTD, showing linear convergence rate and reduced variance and bias errors.
result VRTD converges to a fixed-point solution with reduced variance and bias errors compared to vanilla TD.
Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…
Improved TD learning with tail averaging and regularization achieves optimal convergence rates.
problem Convergence analysis of TD learning with linear function approximation.
method Tail-averaging and regularization applied to TD learning algorithm.
result Achieves optimal O(1/t) convergence rate in expectation and with high probability. In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…
We are interested in understanding stability (almost sure boundedness) of stochastic approximation algorithms (SAs) driven by a `controlled Markov' process. Analyzing this class of algorithms is important, since many reinforcement learning (RL) algorithms can be cast as SAs driven by a `controlled Markov' process. In t…
Proposes a new metric to quantify the difference between neural network representations based on downstream task performance.
problem The lack of a consistent metric to measure the difference between neural network representations.
method Introduced the Transferred Discrepancy (TD) metric, which evaluates the difference between representations based on their performance on downstream tasks.
result TD provides fine-grained information for various downstream tasks and can evaluate the effectiveness of different training strategies.
The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…
This work analyzes actor-critic methods for faster convergence.
problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.
method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.
result Actor-critic method finds a first-order stationary point with ildeO(ε−2.5) sample complexity. The study proposes using TD error for selecting σ in Q(σ, λ).
problem Selecting the value of σ in Q(σ, λ) based on state characteristics.
method TD error as a heuristic for selecting σ.
result TD error can effectively select σ based on state characteristics.
Study on distributional TD learning with linear approximations for better return estimation.
problem Estimating the return distribution of a policy in reinforcement learning.
method Finite-sample analysis of distributional TD learning with linear function approximation, using the linear-categorical Bellman equation and exponential stability arguments for products of random matrices.
result Sample complexity of linear distributional TD learning matches that of classic linear TD learning, indicating similar difficulty in estimating return distribution versus its expectation.
Study on improving the linear two-time-scale stochastic approximation method with a restarting scheme.
problem Characterizing and optimizing the finite-time complexity of linear two-time-scale stochastic approximation.
method Analysis of mean square errors, introduction of a restarting scheme to improve performance.
result The method achieves an exact convergence to the desired solution with improved complexity under time-varying step sizes.
While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known. Here we take a first step towards extending theoretical convergence guarantees to TD learning with nonlinear…
Unified framework for PE and TD methods in continuous time and space.
problem Policy evaluation and TD learning in continuous settings.
method Martingale characterization for designing PE algorithms.
result Convergent time-discretized algorithms converge to continuous-time counterparts.
Aims to describe neural network training dynamics using two-time-scale models.
problem Lack of a general mathematical description of neural network training.
method Introduces a theoretical framework based on two-time-scale population dynamics.
result Derives selection-mutation equations and effective fitness for hyperparameters.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
The problem of on-line off-policy evaluation (OPE) has been actively studied in the last decade due to its importance both as a stand-alone problem and as a module in a policy improvement scheme. However, most Temporal Difference (TD) based solutions ignore the discrepancy between the stationary distribution of the beh…
In this paper, we provide a unified analysis of temporal difference learning algorithms with linear function approximators by exploiting their connections to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in the control community to characterize the exact behaviors of the first and second order …
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.