KF-LAX uses KFAC to improve sample efficiency in reinforcement learning.
problem Sample efficiency and low variance in gradient-based optimization methods.
method Kronecker-factored curvature estimation (KFAC) applied to RELAX gradient estimator.
result Improved performance on synthetic and Atari games.
A quantum reinforcement learning algorithm reduces sample complexity.
problem Quantum reinforcement learning under model-free settings with quantum oracle access.
method Quantum Natural Policy Gradient (QNPG) algorithm replacing random sampling with deterministic gradient estimation.
result QNPG achieves a sample complexity of i l d e O ( ε − 1.5 ) ilde{\mathcal{O}}(ε^{-1.5}) i l d e O ( ε − 1.5 ) for queries to the quantum oracle, significantly improving classical lower bound. Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.
problem Continuous state-action reinforcement learning problems.
method Introduced an alternative policy gradient theorem using weak derivatives.
result The new approach yields algorithms that converge almost surely to stationary points of the value function.
Paper improves reinforcement learning efficiency with deterministic value gradients.
problem High sample complexity in model-free DDPG algorithms for continuous control tasks.
method Proposes DVG and DVPG algorithms with infinite horizon value gradients to improve sample efficiency.
result DVPG algorithm substantially outperforms state-of-the-art methods on continuous control benchmarks.
New gradient estimators simplify policy gradient algorithms in reinforcement learning.
problem Challenges in creating effective gradient estimators for reinforcement learning.
method Total derivative rule and graphical models for gradient estimation.
result New gradient estimators lead to improved performance in reinforcement learning.
New method circumvents non-convexity in bilevel RL via hyper-gradient.
problem Non-convexity in lower-level RL problems in bilevel reinforcement learning.
method Characterizing hyper-gradient via fully first-order information, circumventing convexity assumption.
result Developed model-based and model-free algorithms with convergence rate O ( ε − 1 ) O(ε^{-1}) O ( ε − 1 ) . Integrates multiple feedback channels into policy learning for reinforcement learning.
problem Combining multiple types of feedback into policy gradient algorithms.
method Lagrangian relaxation to satisfy constraints using gradient descent while maximizing rewards.
result Constraints are respected and can accelerate learning in reinforcement learning tasks.
New method improves convergence of RL meta-learning.
problem Improving convergence in model-agnostic meta-reinforcement learning.
method Proposes Stochastic Gradient Meta-Reinforcement Learning (SG-MRL) to find ε ε ε -first-order stationary points. result Derives iteration and sample complexity for SG-MRL.
DPG makes reinforcement learning safer with human advice.
problem Unsafe reinforcement learning in shared environments.
method Extends Policy Gradient to incorporate human directives.
result DPG learns faster and more safely than reward-based methods.
Paper proposes MCTSPO for better reinforcement learning policy optimization.
problem Local optima and saddle points in gradient-based methods and poor initialization in gradient-free methods.
method Monte-Carlo tree search combined with gradient-free optimization.
result Improved performance on reinforcement learning tasks with deceptive or sparse reward functions.
This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-cr…
Improves reinforcement learning by combining off-policy data and exploration.
problem Data inefficiency and local optima in policy gradient methods.
method Combines off-policy data reuse, exploration, and deterministic policies with stochastic optimization.
result Successfully learns solutions using fewer interactions than standard methods.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m − 1 / 2 m^{-1/2} m − 1/2 . Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
A novel meta-learning method using ES for efficient reinforcement learning.
problem Sample inefficiency in reinforcement learning.
method Evolution strategies (ES) for exploration in parameter space, deterministic policy gradients for adaptation.
result Demonstrates improved performance in high-dimensional control tasks compared to gradient-based methods.
FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
Optimized decision trees for reinforcement learning, improving sample complexity and interpretability.
problem Updating decision trees online in reinforcement learning.
method Gradient update over differentiable decision trees, including theoretical justification and empirical validation.
result Our approach outperforms neural networks in sample complexity and achieves higher rewards online.
Policy-gradient method controls multiple non-cohesive targets.
problem Controlling multiple non-cohesive targets in a decentralized manner.
method Proximal Policy Optimization for target selection and driving.
result Effective control of non-cohesive targets without prior dynamics knowledge.
Develops DPG methods for continuous-time RL with deterministic policies.
problem High variance and slow convergence in stochastic policy RL methods.
method Derives continuous-time policy gradient formula and proposes CT-DDPG algorithm.
result CT-DDPG achieves superior stability and faster convergence in continuous-time RL.
Paper introduces a new reinforcement learning method with improved performance.
problem Designing and analyzing efficient reinforcement learning algorithms.
method Proximal gradient temporal difference learning (GTD) with accelerated algorithm GTD2-MP.
result GTD algorithms have linear complexity and improved convergence rate.
Novel algorithm reduces sample complexity for reinforcement learning.
problem Improving sample efficiency in reinforcement learning.
method SRVR-PG algorithm with recursive variance reduction.
result Reduces sample complexity to O ( 1 / ε 3 / 2 ) O(1/ε^{3/2}) O ( 1/ ε 3/2 ) . A novel neural network training method reduces gradient variance for faster and better reinforcement learning.
problem Improving convergence and generalization in deep reinforcement learning.
method Gradient Monitoring (GM) approach to dynamically adjust the learning process based on feedback.
result The proposed methods, especially AM-WGM, significantly enhance model performance and generalization.
This work overcomes bias in concave multi-objective reinforcement learning.
problem Gradient bias in policy gradient methods for concave scalarized multi-objective reinforcement learning.
method Developed a Natural Policy Gradient (NPG) algorithm with a multi-level Monte Carlo (MLMC) estimator.
result Achieved optimal O ~ ( ε − 2 ) \widetilde{\mathcal{O}}(ε^{-2}) O ( ε − 2 ) sample complexity for computing an ε ε ε -optimal policy. This paper uses deep reinforcement learning to optimize stock portfolios considering transaction costs and risks.
problem Optimizing stock portfolios with transaction costs and risks.
method Formulated stock portfolio optimization as a reinforcement learning problem, applied DDPG, GDPG, and PPO algorithms, and used Wavelet Transform.
result DDPG and GDPG algorithms outperformed PPO in continuous action space.
Meta-gradient RL learns to learn from experience.
problem Learning efficiency and adaptability in reinforcement learning.
method Meta-gradient descent to discover its own objective function.
result Meta-gradient RL adapts to learn more efficiently over time.
The paper introduces a health-informed policy gradient method for multi-agent reinforcement learning.
problem Optimizing joint reward functions in multi-agent systems with varying agent health.
method Health-informed credit assignment in a multi-agent proximal policy optimization algorithm.
result Significant improvement in learning performance compared to traditional methods.
Gradient-based meta-RL fails with incorrect task distributions, leading to instability and poor performance.
problem Gradient-based meta-RL's sensitivity to task distributions causes instability and poor performance.
method Proposes meta Active Domain Randomization (meta-ADR) to learn task distributions for gradient-based meta-RL.
result Meta-ADR improves stability and generalization of MAML on simulated locomotion and navigation tasks.
The paper tackles scalarization issues in A2C RL algorithms, proposing methods to avoid gradient overlap and noise.
problem Scalarization issues in A2C RL algorithms leading to gradient overlap and uncontrolled noise.
method Proposes techniques to avoid gradient overlap and noise in A2C RL algorithms.
result Pilot experiments show the proposed method speeds up training in A2C RL algorithms.
A new correction term improves sample efficiency in deep reinforcement learning.
problem Momentum accumulation in TD learning leads to doubly stale gradients.
method Proposed a correction term to address the issue of doubly stale gradients.
result Improves sample efficiency in policy evaluation.
Unified reinforcement learning objective improves policy performance prediction.
problem Improving policy performance prediction in reinforcement learning.
method Proposed a counterfactual objective and an emphatic approach to compute unbiased policy gradients.
result Geoff-PAC algorithm demonstrates empirical success in deep RL benchmarks.
G2Ns combine genetic genes with neural networks for reinforcement learning.
problem Improving sample efficiency and performance in reinforcement learning.
method Genetic-Gated Networks (G2Ns) that integrate binary genetic genes with neural network hidden layers.
result G2Ns achieve significant improvements in sample efficiency and performance in reinforcement learning.
Proof of convergence for multi-objective optimization using inverse reinforcement learning.
problem Proving convergence in multi-objective optimization problems.
method Wasserstein inverse reinforcement learning with projective subgradient method and gradient descent.
result Convergence of inverse reinforcement learning for multi-objective optimization.
Paper presents a rank-1 approximation method for natural policy gradients in deep RL.
problem Computing natural gradients requires inverting the Fisher Information Matrix, which is computationally expensive.
method Develops a rank-1 approximation to the inverse Fisher Information Matrix for efficient natural policy optimization.
result The rank-1 approximation converges faster and has similar sample complexity to stochastic policy gradient methods.
Study policy gradient and actor-critic methods for continuous-time reinforcement learning.
problem Continuous-time reinforcement learning with policy gradient and actor-critic approaches.
method Regularized exploratory formulation, martingale approach, simultaneous policy and value function updates.
result Proposed two types of actor-critic algorithms for online and offline learning.
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
SVRG reduces gradient evaluations for policy evaluation in reinforcement learning.
problem Policy evaluation in reinforcement learning with high computational costs.
method Two variants of SVRG for policy evaluation that reduce gradient calculations.
result Significant reduction in the number of gradient evaluations while preserving linear convergence speed.
A new method reduces communication in distributed RL without sacrificing performance.
problem High communication overhead in distributed RL systems.
method Adaptive policy gradient approach that skips communication during iterations.
result Reduces communication rounds needed for learning accuracy without degrading performance.
Paper uses deep reinforcement learning for cryptocurrency market making.
problem Stochastic inventory control challenges faced by market makers.
method Two policy gradient-based algorithms interact with an environment representing limit order book data and order flow statistics. A forward-feed neural network approximates the policy function, and two reward functions are compared.
result Demonstrates deep reinforcement learning's effectiveness in solving market making challenges.
New methods reduce variance in stochastic computation graph optimization.
problem High variance in gradient estimates from SCGs.
method Introducing value functions, baselines, and critics to derive lower-variance gradient estimates from partial model evaluations.
result Lower-variance gradient estimates from partial model evaluations, making optimization more efficient.
Paper shows RL's policy gradient methods can be viewed as supervised learning problems.
problem Connecting RL and SL approaches.
method Proves PGM as supervised learning problem, replaces true labels with discounted rewards.
result PGM can be seen as supervised learning, emphasizing similarities.
CPGAs converge to locally optimal policies for coagent networks.
problem Training stochastic neural networks using reinforcement learning.
method Proved convergence of CPGAs and extended prior theory to asynchronous and recurrent networks.
result CPGAs converge to locally optimal policies for coagent networks, including asynchronous and recurrent networks.
Paper proposes an ε ε ε -policy gradient for online pricing, reducing regret to O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) .
problem Online pricing learning task
method Combines model-based and model-free reinforcement learning, using ε ε ε -greedy with gradient descent. result Achieves expected regret of order O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) . This paper solves inverse reinforcement learning with options framework.
problem Reinforcement learning in complex environments with expert demonstrations.
method Gradient method for parametrized options, deducing Q-feature and reward feature spaces, optimal reward function selection.
result Recovered rewards provide effective solution to IRL problem and accelerate transfer learning.
New algorithm reduces complexity in multi-agent reinforcement learning.
problem High computational complexity in exact computations for multi-agent reinforcement learning.
method Design of a scalable algorithm based on Natural Policy Gradient, using local information and limited communication.
result Converges to globally optimal policy with dimension-free complexity and localization error.
VRMPO improves reinforcement learning efficiency with stochastic mirror descent.
problem Improving sample efficiency in reinforcement learning.
method VRMPO algorithm with a variance-reduced policy gradient estimator and stochastic mirror descent.
result VRMPO achieves an ε-approximate first-order stationary point with only O(ε^(-3)) sample trajectories.
Proposes a new policy gradient algorithm to improve reinforcement learning efficiency and stability.
problem Inefficiency and instability of DDPG in practical applications, and difficulty in controlling Q estimation bias and variance.
method Introduces a Regularly Updated Deterministic (RUD) policy gradient algorithm.
result The RUD algorithm makes better use of new data and has lower Q value variance, leading to improved performance.
New reinforcement learning algorithms improve policy optimization with entropy regularization.
problem Improving policy optimization in reinforcement learning.
method Soft policy gradient theorem (SPGT) and new policy optimization algorithms.
result New algorithms outperform prior works on various benchmark tasks.