Improved TD learning reduces batch sampling error.
problem TD learning can be inaccurate in batch settings due to action weighting.
method Introduced PSEC-TD(0) to correct action weighting using importance sampling.
result PSEC-TD(0) produces more accurate value function estimates.
Improved TD learning reduces variance and bias errors.
problem Inefficient optimization variance in TD learning.
method Proposed a mathematically solid analysis of VRTD, showing linear convergence rate and reduced variance and bias errors.
result VRTD converges to a fixed-point solution with reduced variance and bias errors compared to vanilla TD.
Unified analysis of TD learning using MJLS theory for linear function approximators.
problem Characterizing the exact behaviors of TD learning algorithms with linear function approximators.
method Exploiting connections to Markov jump linear systems (MJLS) theory to analyze TD learning algorithms.
result Closed-form expressions for mean and covariance matrix of TD estimation error at any time step.
The study proposes using TD error for selecting σ in Q(σ, λ).
problem Selecting the value of σ in Q(σ, λ) based on state characteristics.
method TD error as a heuristic for selecting σ.
result TD error can effectively select σ based on state characteristics.
Improved TD learning with tail averaging and regularization achieves optimal convergence rates.
problem Convergence analysis of TD learning with linear function approximation.
method Tail-averaging and regularization applied to TD learning algorithm.
result Achieves optimal O ( 1 / t ) O(1/t) O ( 1/ t ) convergence rate in expectation and with high probability. New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
problem TD learning's sensitivity to step size specification.
method Implicit TD algorithms reformulate TD updates into fixed point equations.
result Implicit TD algorithms are more stable and less sensitive to step size.
Adaptive TD learning reduces bias in policy evaluation by switching between TD and MC methods.
problem Achieving accurate policy evaluation with Temporal Difference (TD) learning in the presence of state-specific uncertainty.
method Adaptive switching between TD and Monte Carlo (MC) methods, using learned confidence intervals to detect and mitigate bias.
result The proposed adaptive algorithm outperforms existing methods in policy evaluation tasks.
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
Stochastic differential equation approximation for linear TD(0) under Markovian noise
problem Temporal-difference learning with linear function approximation
method Stochastic differential equation approximation
result Explains the constant-stepsize error floor
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD( λ λ λ ). result Improved numerical stability and broader step-size range.
Improves stability of actor-critic methods by penalizing TD error.
problem Instability in actor-critic methods during learning.
method Regularize actor's learning objective by penalizing critic's TD error.
result Improves stability and overall performance of actor-critic methods.
Paper improves TD(0) convergence rate with LFA, i.i.d. samples, and averaging.
problem Improving convergence rate of TD(0) with linear function approximation.
method Polyak-Juditsky averaging, i.i.d. samples, strong mixing assumption.
result Established a new convergence rate for Mean-Square Error (MSE) of approximated function.
TD learning reduces prediction error in Markov chain problems.
problem Estimating value functions in Markov chains with temporal inconsistency.
method Temporal difference learning minimizes temporal inconsistency between successive estimates.
result TD learning can significantly reduce mean-squared error in value estimates.
The paper identifies overfitting as the main bottleneck in efficient deep reinforcement learning.
problem Improving sample efficiency in deep reinforcement learning.
method Empirical analysis on DMC tasks to identify overfitting as the main issue and developing a hill-climbing method targeting validation TD error.
result Overfitting is the primary bottleneck in sample-efficient deep RL, and regularization techniques can control this.
TD-Flow improves long-term predictions in agent learning.
problem Cumulative errors in step-by-step inference of future states.
method Leverages flow-matching techniques and a novel Bellman equation to learn accurate geometric horizon models.
result Significantly reduces errors at long horizons compared to prior methods.
Neural TD and Q-learning prove to converge globally to optimal solutions.
problem Nonconvexity and divergence in neural TD due to value function approximation.
method Proving global convergence of neural TD and Q-learning using overparametrization of neural networks.
result Neural TD and Q-learning converge globally to the global optimum of mean-squared projected Bellman error.
This paper develops source traces for faster TD learning.
problem Improving temporal difference learning speed and generalization.
method Introduces source traces as a backward view of successor representations, enabling TD errors to be propagated to potential causal states.
result Demonstrates faster generalization and improved performance of source traces compared to previous methods.
Study non-parametric value function estimation from a single path.
problem Estimating value function from a single trajectory in Markov reward processes.
method Kernel-based multi-step temporal difference (TD) estimates, including K K K -step look-ahead TD and TD ( λ ) (λ) ( λ ) . result Non-asymptotic guarantees for TD estimates, capturing interactions between mixing time and model mis-specification.
Improved TD learning for non-i.i.d. Markovian data.
problem Convergence analysis of two time-scale TD learning under Markovian samples.
method Non-asymptotic convergence analysis of two time-scale TD with gradient correction under Markovian data.
result Two time-scale TD can converge as fast as O(log t/(t^(2/3))) under diminishing stepsize.
TDS provides exact samples for conditional distributions in diffusion models.
problem Lack of exact sampling methods for diffusion models.
method Sequential Monte Carlo (SMC) algorithm with twisting technique.
result TDS offers more accurate approximations with fewer particles compared to heuristics.
New bounds show linear predictors rarely overfit with certain optimization methods.
problem Bounding test error for linear predictors with stochastic optimization methods.
method Coupling argument for fixed point methods like stochastic and batch mirror descent.
result Locally-adapted rates that depend on predictor properties, not global problem structure.
Study cyber-attacks on RL algorithms, focusing on cost signal manipulation.
problem Adversarial manipulation of cost signals in reinforcement learning.
method Quantitative analysis of TD( λ λ λ ) and Q Q Q -learning algorithms under manipulation. result Bound on approximation error for TD( λ λ λ ) and convergence properties for Q Q Q -learning under stealthy attacks. Temporal Difference learning or TD( λ λ λ ) is a fundamental algorithm in the field of reinforcement learning. However, setting TD's λ λ λ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the λ λ λ selection problem as a bias-variance trade-off where the solution is …
Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.
problem Inaccurate error representations and compromised uncertainty estimation in conventional uncertainty-aware TD learning.
method Introduces a novel framework for generalized Gaussian error modeling in deep reinforcement learning, incorporating higher-order moments, particularly kurtosis, to improve uncertainty estimation and mitigation.
result Significant performance gains in policy gradient algorithms with the proposed framework.
TD learning with neural networks can lead to worse solutions than Monte-Carlo methods, especially in discontinuous value functions.
problem TD learning with neural networks can propagate approximation errors, leading to worse solutions than Monte-Carlo methods.
method Investigated the issue of approximation errors in areas of sharp discontinuities of the value function being further propagated by bootstrap updates.
result Empirical and analytical evidence shows that leakage propagation occurs in TD learning with function approximation, especially in sharp discontinuities.
Paper improves TD learning algorithm bounds with linear approx.
problem Sharp bounds for TD method performance in MDPs.
method Polyak-Ruppert averaging, universal step size, refined error bounds, stability of random matrices.
result Near-optimal variance and bias terms achieved.
Proposes a method to use λ λ λ -returns in replay-based reinforcement learning.
problem Difficulty in implementing λ λ λ -returns in off-policy methods with experience replay. method Promotes short sequences into a cache, precomputes λ λ λ -returns, and uses TD errors for dynamic λ λ λ -values. result Enhances DQN performance on Atari games, even under partial observability.
New robust TD learning method for critical domains without observing rare events.
problem Learning robust policies in critical domains with rare events.
method Introduces a κ κ κ -operator for robust TD learning, proving convergence and demonstrating superior performance. result Empirical evaluations show superior performance and robustness to small model errors.
TD learning becomes a gradient descent for reversible policies, improving reinforcement learning stability.
problem TD learning's convergence issues in large or infinite state spaces with nonlinear approximations.
method Proving approximate TD is a gradient descent for reversible policies, even with nonlinear approximations.
result Approximate TD is a gradient descent for reversible policies, improving reinforcement learning stability.
Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.
problem Learning effective control policies from noisy signals.
method Combines True Online Sarsa( λ λ λ ) with step-size optimization and decay. result Swift-Sarsa learns relevant signals without prior knowledge.
Paper analyzes biased stochastic approximation with a novel multistep Lyapunov function.
problem Finite-time analysis of biased stochastic approximation algorithms.
method Developed a multistep Lyapunov function to analyze convergence and error bounds.
result First finite-time error bounds for TD- and Q-learning with linear function approximation.
The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
Unified framework for PE and TD methods in continuous time and space.
problem Policy evaluation and TD learning in continuous settings.
method Martingale characterization for designing PE algorithms.
result Convergent time-discretized algorithms converge to continuous-time counterparts.
RO-TD learns sparse value functions efficiently.
problem Learning sparse value functions efficiently.
method RO-TD integrates off-policy convergent gradient TD methods and online convex regularization.
result RO-TD learns sparse value functions with low computational complexity.
Study shows TD(0) with linear approx. converges for reversible Markov chains.
problem TD(0) divergence with off-policy and function approximation.
method Analyzes standard TD(0) with reversible Markov chains, adapting stochastic approximation framework.
result Establishes convergence with probability one for projected Bellman error = 0.
Introduces TDRC to balance TD's ease and soundness.
problem TD learning's instability and divergence issues.
method Gradient Temporal-Difference Learning with Regularized Corrections (TDRC).
result TDRC performs as well as TD when TD works, but is sound in divergent cases.
New target-based TD learning algorithms improve deep Q-learning convergence.
problem Improving convergence of deep Q-learning algorithms.
method Introducing averaging TD, double TD, and periodic TD algorithms.
result Established asymptotic convergence analyses for averaging TD and double TD, and finite sample analysis for periodic TD.
Combines Hebbian and DQN for better POMDP problem solving.
problem Difficult POMDP problems with TD errors.
method Modulated Hebbian plus Q network architecture (MOHQA) integrating Hebbian and DQN.
result Improved DQN performance and outperformed other algorithms on some POMDPs.
Uniform TD(0) bound derived for function approximation with Markov noise.
problem Uniform concentration bound for TD(0) with function approximation.
method Contractive stochastic approximation, martingale and Markov noises, Poisson equation, relaxed concentration inequalities.
result Uniform all-time concentration bound for TD(0) with linear function approximation.
Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a γ \sqrtγ γ -contraction modulus (where γ γ γ is the …
Proposes a new sampling method for deep Q-learning to improve efficiency and convergence.
problem Challenges in learning state-action value function from replay buffer.
method State distribution-aware sampling method to balance replay times for transitions.
result Reduces unnecessary TD updates and increases updates for uncertain state-action values.
Improved TD(0) algorithm for reinforcement learning with linear approximations.
problem Sensitivity of TD(0) to stepsizes and slow convergence.
method AdaTD(0) - Adaptive projected variant of TD(0) with linear function approximation.
result AdaTD(0) converges faster and is less sensitive to stepsizes.
Decentralized TD learning converges linearly with linear function approximation.
problem Policy evaluation in fully decentralized multi-agent reinforcement learning.
method Temporal-difference learning with linear function approximation, analyzing i.i.d. and Markovian samples.
result Local estimates converge linearly to the optimum under both i.i.d. and Markovian samples.
In reinforcement learning, the TD( λ λ λ ) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD( λ λ λ ) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…
Paper analyzes SA for fixed-point equations with noise, establishing convergence rates.
problem Solving fixed-point equations with noisy data.
method Uses smooth convex envelopes to construct Lyapunov functions and show negative drift.
result Establishes first-known convergence rate for V-trace algorithm in RL.
This paper re-evaluates TD in deep RL, finding MC can be a viable alternative.
problem Understanding the role of temporal differencing (TD) in deep reinforcement learning.
method Designed environments to control for factors affecting performance in deep RL, comparing TD with infinite-horizon Monte Carlo (MC).
result Finite-horizon Monte Carlo is not inferior to TD, even with sparse or delayed rewards.
New STDP rule for spiking neurons solves discrete action reinforcement learning tasks.
problem Applying standard STDP to discrete action reinforcement learning tasks.
method Feedback-modulated TD-STDP learning rule for spiking neuron networks.
result Feedback modulation improves credit assignment in reinforcement learning.