Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.
problem Continuous state-action reinforcement learning problems.
method Introduced an alternative policy gradient theorem using weak derivatives.
result The new approach yields algorithms that converge almost surely to stationary points of the value function.
Integrates multiple feedback channels into policy learning for reinforcement learning.
problem Combining multiple types of feedback into policy gradient algorithms.
method Lagrangian relaxation to satisfy constraints using gradient descent while maximizing rewards.
result Constraints are respected and can accelerate learning in reinforcement learning tasks.
Improves reinforcement learning by combining off-policy data and exploration.
problem Data inefficiency and local optima in policy gradient methods.
method Combines off-policy data reuse, exploration, and deterministic policies with stochastic optimization.
result Successfully learns solutions using fewer interactions than standard methods.
Develops DPG methods for continuous-time RL with deterministic policies.
problem High variance and slow convergence in stochastic policy RL methods.
method Derives continuous-time policy gradient formula and proposes CT-DDPG algorithm.
result CT-DDPG achieves superior stability and faster convergence in continuous-time RL.
Policy-gradient method controls multiple non-cohesive targets.
problem Controlling multiple non-cohesive targets in a decentralized manner.
method Proximal Policy Optimization for target selection and driving.
result Effective control of non-cohesive targets without prior dynamics knowledge.
Unified reinforcement learning objective improves policy performance prediction.
problem Improving policy performance prediction in reinforcement learning.
method Proposed a counterfactual objective and an emphatic approach to compute unbiased policy gradients.
result Geoff-PAC algorithm demonstrates empirical success in deep RL benchmarks.
Paper proposes MCTSPO for better reinforcement learning policy optimization.
problem Local optima and saddle points in gradient-based methods and poor initialization in gradient-free methods.
method Monte-Carlo tree search combined with gradient-free optimization.
result Improved performance on reinforcement learning tasks with deceptive or sparse reward functions.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m − 1 / 2 m^{-1/2} m − 1/2 . A quantum reinforcement learning algorithm reduces sample complexity.
problem Quantum reinforcement learning under model-free settings with quantum oracle access.
method Quantum Natural Policy Gradient (QNPG) algorithm replacing random sampling with deterministic gradient estimation.
result QNPG achieves a sample complexity of i l d e O ( ε − 1.5 ) ilde{\mathcal{O}}(ε^{-1.5}) i l d e O ( ε − 1.5 ) for queries to the quantum oracle, significantly improving classical lower bound. New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
Entropy regularization improves policy optimization in reinforcement learning.
problem Improving policy optimization in reinforcement learning.
method Entropy regularization is introduced to soften the greedy policy towards a more diverse softmax policy, leading to a continuously parameterized algorithm that interpolates between policy gradient and Q-learning.
result An intermediate algorithm can improve performance in reinforcement learning.
FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
New reinforcement learning algorithms improve policy optimization with entropy regularization.
problem Improving policy optimization in reinforcement learning.
method Soft policy gradient theorem (SPGT) and new policy optimization algorithms.
result New algorithms outperform prior works on various benchmark tasks.
The paper introduces a health-informed policy gradient method for multi-agent reinforcement learning.
problem Optimizing joint reward functions in multi-agent systems with varying agent health.
method Health-informed credit assignment in a multi-agent proximal policy optimization algorithm.
result Significant improvement in learning performance compared to traditional methods.
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
New gradient estimators simplify policy gradient algorithms in reinforcement learning.
problem Challenges in creating effective gradient estimators for reinforcement learning.
method Total derivative rule and graphical models for gradient estimation.
result New gradient estimators lead to improved performance in reinforcement learning.
Novel algorithm reduces sample complexity for reinforcement learning.
problem Improving sample efficiency in reinforcement learning.
method SRVR-PG algorithm with recursive variance reduction.
result Reduces sample complexity to O ( 1 / ε 3 / 2 ) O(1/ε^{3/2}) O ( 1/ ε 3/2 ) . Study policy gradient and actor-critic methods for continuous-time reinforcement learning.
problem Continuous-time reinforcement learning with policy gradient and actor-critic approaches.
method Regularized exploratory formulation, martingale approach, simultaneous policy and value function updates.
result Proposed two types of actor-critic algorithms for online and offline learning.
This paper uses deep reinforcement learning to optimize stock portfolios considering transaction costs and risks.
problem Optimizing stock portfolios with transaction costs and risks.
method Formulated stock portfolio optimization as a reinforcement learning problem, applied DDPG, GDPG, and PPO algorithms, and used Wavelet Transform.
result DDPG and GDPG algorithms outperformed PPO in continuous action space.
VRMPO improves reinforcement learning efficiency with stochastic mirror descent.
problem Improving sample efficiency in reinforcement learning.
method VRMPO algorithm with a variance-reduced policy gradient estimator and stochastic mirror descent.
result VRMPO achieves an ε-approximate first-order stationary point with only O(ε^(-3)) sample trajectories.
SVRG reduces gradient evaluations for policy evaluation in reinforcement learning.
problem Policy evaluation in reinforcement learning with high computational costs.
method Two variants of SVRG for policy evaluation that reduce gradient calculations.
result Significant reduction in the number of gradient evaluations while preserving linear convergence speed.
This work uses MMD to find diverse policies in reinforcement learning.
problem Finding multiple near-optimal policies for a task.
method Formalizes policy difference as trajectory distribution discrepancy, uses MMD for optimization.
result Derives gradient-based optimization for diverse policy identification.
The paper develops a safe policy gradient algorithm for reinforcement learning.
problem Safety issues in reinforcement learning for real-world control tasks.
method Stochastic optimization perspective, meta-parameter schedules, adaptive selection of step size and batch size.
result Monotonic improvement guarantees for a wide class of parametric policies.
Paper presents a rank-1 approximation method for natural policy gradients in deep RL.
problem Computing natural gradients requires inverting the Fisher Information Matrix, which is computationally expensive.
method Develops a rank-1 approximation to the inverse Fisher Information Matrix for efficient natural policy optimization.
result The rank-1 approximation converges faster and has similar sample complexity to stochastic policy gradient methods.
Improves reinforcement learning policies for robustness.
problem Lack of robustness in reinforcement learning policies.
method Risk-aware Distributional Reinforcement Learning (SDPG) with CVaR.
result Risk-averse policies achieve robustness against disturbances.
We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gradients. We empirically show that, despite its simplicity in implementation, entropy regularization c…
This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
problem The bias-variance tradeoff in estimating and using policy gradients for deep reinforcement learning.
method Introduces elite policy gradients and a two-step merging method to balance bias-variance tradeoffs.
result Two-step merging outperforms interpolation merging and state-of-the-art algorithms on benchmark control tasks.
CPGAs converge to locally optimal policies for coagent networks.
problem Training stochastic neural networks using reinforcement learning.
method Proved convergence of CPGAs and extended prior theory to asynchronous and recurrent networks.
result CPGAs converge to locally optimal policies for coagent networks, including asynchronous and recurrent networks.
Paper improves policy updates in reinforcement learning to speed up learning.
problem Slow learning and unlearning in policy optimization.
method Introduces a novel gradient update and a modified policy update.
result Proves modified policy update converges to global optimality.
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Paper improves reinforcement learning efficiency with deterministic value gradients.
problem High sample complexity in model-free DDPG algorithms for continuous control tasks.
method Proposes DVG and DVPG algorithms with infinite horizon value gradients to improve sample efficiency.
result DVPG algorithm substantially outperforms state-of-the-art methods on continuous control benchmarks.
New method improves convergence of RL meta-learning.
problem Improving convergence in model-agnostic meta-reinforcement learning.
method Proposes Stochastic Gradient Meta-Reinforcement Learning (SG-MRL) to find ε ε ε -first-order stationary points. result Derives iteration and sample complexity for SG-MRL.
Introduces GAMPS for better model-based policy learning.
problem Misspecified model classes lead to poor policy estimates.
method Exploits current policy to learn approximate transition model, focusing on relevant parts of the environment.
result Empirically validated GAMPS on benchmark domains, demonstrating improved properties.
A new method for learning policies in multiple environments.
problem Learning policies that work in different but related environments.
method Decentralized entropy-regularized policy gradient method.
result The method can learn effective policies in multiple environments.
A new method reduces communication in distributed RL without sacrificing performance.
problem High communication overhead in distributed RL systems.
method Adaptive policy gradient approach that skips communication during iterations.
result Reduces communication rounds needed for learning accuracy without degrading performance.
A new method for RL with continuous actions improves stability and scalability.
problem Stability and scalability issues in existing RL methods.
method Soft policy gradient with entropy regularization, combined with double sampling for soft Bellman equation.
result Outperforms off-policy prior methods in continuous action RL tasks.
This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-cr…
Unified treatment of reinforcement learning via convex duality.
problem Applying Fenchel-Rockafellar duality to reinforcement learning.
method Unified derivation of various RL settings using convex duality.
result Ability to perform policy evaluation and on-policy policy gradient with offline data.
Paper tackles efficient policy gradient estimation from off-policy data.
problem Estimating policy gradients from off-policy data is challenging and inefficient.
method Derives asymptotic lower bounds, proposes a meta-algorithm with 3-way robustness, and establishes convergence guarantees.
result Meta-algorithm achieves the lower bound on mean-squared error without parametric assumptions.
This paper tackles robust control of LQR systems with multiplicative noise using policy gradient methods.
problem Robustness in reinforcement learning control of complex systems with multiplicative noise.
method Policy gradient algorithms with gradient domination property for non-convex cost functions.
result Global convergence of policy gradient algorithms to the globally optimum control policy.
Paper proposes an ε ε ε -policy gradient for online pricing, reducing regret to O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) .
problem Online pricing learning task
method Combines model-based and model-free reinforcement learning, using ε ε ε -greedy with gradient descent. result Achieves expected regret of order O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) . Proposes a new policy gradient algorithm to improve reinforcement learning efficiency and stability.
problem Inefficiency and instability of DDPG in practical applications, and difficulty in controlling Q estimation bias and variance.
method Introduces a Regularly Updated Deterministic (RUD) policy gradient algorithm.
result The RUD algorithm makes better use of new data and has lower Q value variance, leading to improved performance.
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
PPG separates policy and value function training phases for better reinforcement learning efficiency.
problem Challenges in traditional reinforcement learning methods for policy and value function optimization.
method Integrates Phasic Policy Gradient framework that splits policy and value function training into distinct phases.
result Significantly improves sample efficiency on Procgen Benchmark compared to PPO.
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
New algorithm reduces complexity in multi-agent reinforcement learning.
problem High computational complexity in exact computations for multi-agent reinforcement learning.
method Design of a scalable algorithm based on Natural Policy Gradient, using local information and limited communication.
result Converges to globally optimal policy with dimension-free complexity and localization error.
Paper shows RL's policy gradient methods can be viewed as supervised learning problems.
problem Connecting RL and SL approaches.
method Proves PGM as supervised learning problem, replaces true labels with discounted rewards.
result PGM can be seen as supervised learning, emphasizing similarities.