Temporal difference learning explained through gradient splitting, improving convergence times.
problem Learning value functions in Markov Decision Processes with linear approximations.
method Interpreting TD learning as gradient splitting and applying convergence proofs from gradient descent.
result Improved convergence times for TD learning, especially with a minor variation.
Paper introduces a new reinforcement learning method with improved performance.
problem Designing and analyzing efficient reinforcement learning algorithms.
method Proximal gradient temporal difference learning (GTD) with accelerated algorithm GTD2-MP.
result GTD algorithms have linear complexity and improved convergence rate.
A new correction term improves sample efficiency in deep reinforcement learning.
problem Momentum accumulation in TD learning leads to doubly stale gradients.
method Proposed a correction term to address the issue of doubly stale gradients.
result Improves sample efficiency in policy evaluation.
FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Differ…
Introduces TDRC to balance TD's ease and soundness.
problem TD learning's instability and divergence issues.
method Gradient Temporal-Difference Learning with Regularized Corrections (TDRC).
result TDRC performs as well as TD when TD works, but is sound in divergent cases.
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
New method makes reinforcement learning robust to heavy-tailed rewards.
problem Heavy-tailed rewards cause statistical outliers in reinforcement learning.
method Dynamic gradient clipping in TD learning and NAC.
result Provably robust TD and NAC achieve optimal sample complexities.
TD learning reduces interference, leading to better generalization.
problem Understanding and reducing interference in TD learning for better generalization.
method Analyzing the inner product of gradients as interference, comparing TD and supervised learning, and examining the dynamics of interference and bootstrapping.
result TD learning leads to low-interference, under-generalizing parameters, while supervised learning does the opposite.
Solves POMDPs with recurrent neural networks and natural policy gradient.
problem Non-stationarity in optimal policies of POMDPs.
method Integrates recurrent neural networks into natural policy gradient and temporal difference learning.
result Non-asymptotic theoretical guarantees for global optimality up to function approximation.
Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that…
Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function V. Two novel algorithms are proposed to approximate the true value…
New algorithms for collaborative reinforcement learning with limited communication.
problem Efficiently learning value functions in multi-agent systems with strict information constraints.
method Distributed gradient-based temporal difference algorithms with consensus schemes.
result Parameter estimates converge to ODEs with defined invariant sets under general assumptions.
Temporal-difference and Q-learning learn feature representations that converge to optimal ones.
problem Understanding how feature representations evolve in temporal-difference and Q-learning with neural networks.
method Mean-field theory applied to overparameterized two-layer neural networks.
result The feature representation converges to the optimal one, generalizing previous results.
Study on reinforcement learning dynamics using statistical physics.
problem Understanding how reinforcement learning dynamics interact with parameters and state features.
method Statistical physics concepts applied to temporal difference learning with linear function approximators.
result Stochastic semi-gradient noise leads to significant plateaus in value error.
Study uses online bootstrap for RL inference, showing effectiveness.
problem Statistical inference for RL parameters in online settings.
method Online bootstrap method applied to TD and GTD algorithms in RL.
result Method is distributionally consistent for policy evaluation inference.
Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is one of the most widely used algorithms in reinforcement learning, its theoretical analysis has proved challenging and few guarantees on its s…
We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distributional}. In the policy evaluation setting, we design two new algorithms called distributional GTD2 a…
TD-Flow improves long-term predictions in agent learning.
problem Cumulative errors in step-by-step inference of future states.
method Leverages flow-matching techniques and a novel Bellman equation to learn accurate geometric horizon models.
result Significantly reduces errors at long horizons compared to prior methods.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m−1/2. The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…
Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.
problem Inaccurate error representations and compromised uncertainty estimation in conventional uncertainty-aware TD learning.
method Introduces a novel framework for generalized Gaussian error modeling in deep reinforcement learning, incorporating higher-order moments, particularly kurtosis, to improve uncertainty estimation and mitigation.
result Significant performance gains in policy gradient algorithms with the proposed framework.
Unified framework for PE and TD methods in continuous time and space.
problem Policy evaluation and TD learning in continuous settings.
method Martingale characterization for designing PE algorithms.
result Convergent time-discretized algorithms converge to continuous-time counterparts.
In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…
PBVFs generalize across policies using learned value functions.
problem RL algorithms forget information about old policies when updating value functions to track the learned policy.
method Introduce Parameter-Based Value Functions (PBVFs) that include policy parameters in their inputs, enabling them to generalize across different policies.
result PBVFs enable zero-shot learning of new policies that outperform any policy seen during training.
Efficient spatio-temporal Gaussian process inference method.
problem Scalable Gaussian process inference for multivariate, spatio-temporal data.
method Combines spatio-temporal filtering with natural gradient variational inference, resulting in a scalable non-conjugate GP method.
result Linear scaling with respect to time and logarithmic scaling with respect to time steps.
Paper presents a new method for Bayesian deep learning that scales to Atari games.
problem Training neural networks on complex environments like Atari games is challenging.
method Adapted temporal difference Q-learning to work with Bayesian inference.
result TAGI allows for analytical inference of neural network parameters, achieving performance comparable to gradient-based methods.
TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.
problem Improving performance of adaptive optimizers in Deep RL.
method TDprop computes per-parameter learning rates based on Jacobi preconditioning of the TD update rule.
result TDprop matches or exceeds Adam's performance in Deep RL experiments, suggesting Jacobi preconditioning can improve adaptive methods.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
Paper analyzes CLT for TTSA with Markovian noise, broadening its applications.
problem Analyzing asymptotic behavior of TTSA under Markovian noise.
method Central Limit Theorem applied to TTSA with Markovian noise.
result Uncovered coupled dynamics of TTSA influenced by Markov chain.
New framework for modular reinforcement learning reduces sample complexity.
problem Achieving independent credit assignment in reinforcement learning.
method Defining modular credit assignment as minimizing algorithmic mutual information, introducing modularity criterion for causal analysis.
result Single-step temporal difference action-value methods meet the modularity criterion, improving sample efficiency.
Temporal Difference Learning analysis under non-i.i.d. data and nonlinear approximation.
problem Finite-sample behavior of TD(0) under non-i.i.d. data and nonlinear approximation.
method High-probability, finite-sample analysis of vanilla TD(0) on polynomially mixing Markov data, assuming Holder continuity and bounded generalized gradients.
result Bounds on the convergence rate of TD(0) with high probability, matching known i.i.d. rates and holding even with nonstationary initialization.
New algorithm borrows future randomness to stabilize model-free control.
problem Double sampling problem in model-free control with nonlinear approximations.
method Borrowing from the future (BFF) algorithm to approximate re-sampling of next states.
result BFF is close to unbiased SGD under smooth dynamics, validated by simulations.
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …
Accurately predicting customer churn using large scale time-series data is a common problem facing many business domains. The creation of model features across various time windows for training and testing can be particularly challenging due to temporal issues common to time-series data. In this paper, we will explore …
New algorithm predicts spatio-temporal events with improved accuracy.
problem Non-stationary spatio-temporal prediction on dense and sparse sequences.
method Probabilistic approach using point processes and self-organizing decision trees.
result Significant performance improvements over baseline and state-of-the-art methods.
Unified analysis of stochastic iterative algorithms using Lyapunov functions.
problem Analyzing convergence of stochastic iterative algorithms for fixed-point equations.
method Lyapunov-based techniques for finite-time analysis of stochastic approximation algorithms.
result Unified mean-square convergence guarantees for various algorithms.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
Combines CNN and LSTM for spatio-temporal graph networks.
problem Improving spatio-temporal feature extraction.
method Proposes a new architecture combining CNN and LSTM temporal blocks.
result Empirical comparison shows our model outperforms existing models.
New loss function equivalence reveals PER's uniform sampling can be improved.
problem Improving Prioritized Experience Replay (PER) for better learning efficiency.
method Transforming non-uniformly sampled data loss functions into uniformly sampled ones.
result Some environments can replace PER with a new loss function without performance loss.
Novel framework for spatio-temporal event analysis using Hawkes processes.
problem Inference of dynamics in spatio-temporal event sequences.
method Randomized Fourier feature-based transformations and gradient descent.
result Improved fitting capability in synthetic and real datasets.
This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-cr…
A simple GI loss improves temporal generalization without complex methods.
problem Temporal drift between train and test distributions in evolving data.
method Gradient Interpolation (GI) loss to regularize temporal complexity.
result GI loss outperforms complex methods on real-world datasets.
Networks are a fundamental tool for modeling complex systems in a variety of domains including social and communication networks as well as biology and neuroscience. Small subgraph patterns in networks, called network motifs, are crucial to understanding the structure and function of these systems. However, the role of…
This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional grad…
New method adapts to unknown mixing time in stochastic optimization.
problem Optimizing with Markovian data where mixing time is unknown.
method Combines MLMC gradient estimation with adaptive learning.
result Achieves optimal convergence rate for convex problems.
Modular pipeline improves stock portfolio prediction robustness under regime changes.
problem Overfitting in deep learning models for non-stationary datasets.
method Modular machine learning pipeline with GBDT models and online learning techniques.
result GBDT models with dropout show high performance, robustness, and generalisability.
Proposes a differentiable STFT for more efficient optimization of hop length.
problem Efficient optimization of hop length in STFT for better temporal control.
method Introduces a differentiable version of STFT with continuous hop length.
result Improves optimization methods like gradient descent for STFT.