New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Algorithm learns optimal policies from diverse data.
problem Optimizing policies from multiple individuals' data.
method Batch, off-policy actor-critic approach.
result Improves learning from diverse training data.
Paper develops a new multi-agent reinforcement learning algorithm.
problem Improving policies in a network of communicating agents.
method Develops a multi-agent off-policy actor-critic algorithm using emphatic temporal difference learning.
result Proves convergence of the algorithm under linear function approximation.
Paper explores using action-value gradients for policy improvement in off-policy actor-critic methods.
problem Improving policies using action-value gradients in off-policy stochastic actor-critic methods.
method Discusses and analyzes the use of action-value gradients for policy improvement, and proposes an incremental approach.
result Demonstrates the feasibility and incremental approach for following the policy gradient.
Soft Actor-Critic improves deep RL with entropy maximization.
problem High sample complexity and brittle convergence in deep RL.
method Maximum entropy reinforcement learning framework, off-policy updates, stochastic actor-critic.
result State-of-the-art performance on continuous control tasks.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.
Paper introduces a meta-critic for accelerating off-policy actor-critic learning.
problem Improving sample efficiency in continuous control tasks.
method Meta-critic that meta-learns an additional loss for the actor.
result Online meta-critic learning leads to improved performance in various continuous control environments.
Paper provides convergence guarantees for off-policy NAC with finite sample complexity.
problem Convergence analysis of off-policy natural actor-critic algorithm.
method Finite-sample analysis with Importance Sampling and Q-trace algorithm.
result Converges to global optimal policy with sample complexity O(ε−3log2(1/ε)). Enhances SAC for better sample efficiency in continuous-action tasks.
problem Improving sample efficiency in soft actor-critic algorithms.
method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).
result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.
A new theorem and algorithm solve off-policy policy gradient problems.
problem Solving the theoretical gap in off-policy policy gradient methods.
method Introduced an off-policy policy gradient theorem using emphatic weightings and developed the ACE algorithm.
result Demonstrated ACE finds the optimal solution in off-policy learning, unlike previous methods.
COF-PAC converges with novel critic and learning method.
problem Convergent off-policy actor-critic with function approximation.
method Two-timescale approach with Gradient Emphasis Learning (GEM).
result First provably convergent COF-PAC with linear critics and nonlinear actor.
New algorithm reduces bias in off-policy reinforcement learning.
problem Challenges in designing off-policy reinforcement learning algorithms.
method Doubly robust off-policy actor-critic (DR-Off-PAC) with a single timescale structure.
result Establishes the first overall sample complexity analysis for a single time-scale off-policy AC algorithm.
Distributed learning method for multi-agent reinforcement learning with policy coordination.
problem Solving multi-agent reinforcement learning problems with coordination.
method Distributed off-policy actor critic with policy consensus.
result The proposed algorithm achieves asymptotic agreement on the global optimal policy function.
Unified reinforcement learning objective improves policy performance prediction.
problem Improving policy performance prediction in reinforcement learning.
method Proposed a counterfactual objective and an emphatic approach to compute unbiased policy gradients.
result Geoff-PAC algorithm demonstrates empirical success in deep RL benchmarks.
New algorithm BEAR reduces instability in off-policy Q-learning.
problem High sensitivity of off-policy Q-learning methods to data distribution.
method Identified and mitigated bootstrapping error through constrained action selection.
result BEAR algorithm learns robustly from various off-policy distributions.
Improves off-policy RL stability with RIS.
problem Stability issues in off-policy RL due to distributional mismatch.
method Relative Importance Sampling (RIS) for off-policy actor-critic.
result RIS stabilizes RL learning by reducing variance.
SAC improves deep RL by balancing reward and randomness.
problem High sample complexity and hyperparameter brittleness in RL.
method Maximum entropy framework, constrained optimization, temperature tuning.
result SAC achieves state-of-the-art performance and stability.
PBVFs generalize across policies using learned value functions.
problem RL algorithms forget information about old policies when updating value functions to track the learned policy.
method Introduce Parameter-Based Value Functions (PBVFs) that include policy parameters in their inputs, enabling them to generalize across different policies.
result PBVFs enable zero-shot learning of new policies that outperform any policy seen during training.
Combines soft greediness with Modified Policy Iteration for more efficient deep reinforcement learning.
problem Improving sample efficiency in deep reinforcement learning.
method Combines soft greediness with Modified Policy Iteration (MPI) for off-policy learning.
result The proposed algorithm is more sample efficient than original PPO and competitive with SAC.
SIL learns to repeat past good decisions for better exploration.
problem Improving exploration in reinforcement learning environments.
method Self-Imitation Learning (SIL) is a simple off-policy actor-critic algorithm.
result SIL significantly improves A2C on Atari games and is competitive with state-of-the-art methods.
AEA dynamically aggregates ensemble targets for actor-critic learning.
problem Static ensemble aggregation methods struggle with overestimation bias and variance.
method Adaptive Ensemble Aggregation (AEA) dynamically constructs ensemble-based targets.
result AEA converges to optimal variance reduction and maximal Fisher information.
New actor-critic algorithm achieves optimal sample efficiency in RL.
problem Achieving ε-optimal policies with minimal samples in RL. method Integrates optimism, off-policy critic estimation, and rare-switching policy resets.
result Sample complexity of O(dH5log∣A∣/ε2+dH4log∣F∣/ε2) trajectories. Paper achieves ε−2 sample complexity for actor-critic methods with minimal assumptions.
problem Achieving ε−2 sample complexity for actor-critic methods under minimal assumptions. method Single-loop, single-timescale implementation; coupled Lyapunov drift framework.
result First ildeO(ε−2) sample complexity guarantee for finding an ε-optimal policy. Automatically finds strong neural network topologies for continuous control tasks.
problem Handcrafted neural network architectures limit the performance of Deep Reinforcement Learning.
method Combines Neuroevolution with off-policy training and proposes a novel architecture mutation operator.
result The proposed Actor-Critic Neuroevolution algorithm often outperforms strong baseline methods.
Improved exploration in SAC using Normalizing Flows policies.
problem Brittleness and inefficiency of DRL algorithms in continuous action spaces.
method Introducing Normalizing Flow policies within the SAC framework to learn more expressive policies.
result Increased stability and better exploration in sparse reward settings.
Optimistic actor-critic tackles linear MDPs with parametric policies.
problem Theoretical limitations of existing actor-critic methods for linear MDPs.
method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.
result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.
New algorithm reduces sample inefficiency and reward bias in AI learning.
problem Implicit reward bias and high sample inefficiency in AI learning.
method Discriminator-Actor-Critic using off-policy Reinforcement Learning.
result Average 10x reduction in policy-environment interaction samples.
SUNRISE improves off-policy RL algorithms by integrating ensemble methods.
problem Stability and exploration issues in off-policy RL algorithms.
method SUNRISE combines ensemble-based weighted Bellman backups and upper-confidence bounds for efficient exploration.
result SUNRISE improves the performance of off-policy RL algorithms across various domains.
New algorithms improve deep RL with entropy-based action selection and environment exploration.
problem Improving deep reinforcement learning algorithms for better sample efficiency and effectiveness.
method Proposes Tsallis entropy Actor-Critic (TAC), Renyi entropy Actor-Critic (RAC), and Ensemble Actor-Critic (EAC) algorithms.
result Empirically, TAC, RAC, and EAC outperform SAC and other algorithms in benchmark control tasks.
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
A new method for continuous control avoids local movement issues.
problem Limitations of policy gradient methods in continuous control.
method Distributional framework and Generative Actor Critic (GAC) method.
result GAC outperforms policy gradient methods in continuous domains.
Self-Tuning Actor-Critic improves reinforcement learning performance.
problem Manual hyperparameter tuning is time-consuming and domain-specific.
method Uses metagradients for online hyperparameter adaptation.
result Improves performance across various domains and tasks.
SAC-NF improves RL exploration by using normalizing flows to discover policies faster.
problem Discovering optimal policies in sparse reward domains.
method Extending SAC with normalizing flows to improve exploration efficiency.
result SAC-NF accelerates policy discovery with smaller parameter usage.
Combines policy gradient and Q-learning for improved data efficiency and stability.
problem Improving data efficiency and stability in reinforcement learning.
method Combines policy gradient with off-policy Q-learning using a replay buffer and fixed points of policy gradient.
result Achieved performance exceeding A3C and Q-learning on Atari games.
This study improves convergence of two-timescale SA under Markovian noise in reinforcement learning.
problem Stability and convergence of two-timescale stochastic approximations under Markovian noise.
method Introduced a new control strategy for the fast timescale parameter.
result Established almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
A streamlined DRL algorithm improves sample efficiency without entropy maximization.
problem Improving sample efficiency in off-policy DRL algorithms.
method Output normalization and non-uniform sampling.
result Proposed algorithm matches SAC's performance without entropy maximization and improves sample efficiency.
New reinforcement learning bound improves generalization for sequential data.
problem Challenges in obtaining generalization guarantees for reinforcement learning due to sequential data.
method PAC-Bayesian reinforcement learning with consideration of Markov dependencies and mixing time.
result Demonstrated practical utility through PB-SAC, providing meaningful confidence certificates.
One-step policy improvement outperforms iterative RL methods on D4RL.
problem Improving offline RL without off-policy evaluation.
method One-step constrained/regularized policy improvement using on-policy Q estimates.
result One-step algorithm outperforms iterative algorithms on D4RL benchmark.
The H∞ control design problem is considered for nonlinear systems with unknown internal system model. It is known that the nonlinear H∞ control problem can be transformed into solving the so-called Hamilton-Jacobi-Isaacs (HJI) equation, which is a nonlinear partial differential equation that is genera…
PCL connects value and policy RL, improving performance across benchmarks.
problem Improving reinforcement learning performance by bridging value and policy approaches.
method Developed Path Consistency Learning (PCL) algorithm that minimizes soft consistency error.
result PCL significantly outperforms actor-critic and Q-learning baselines across benchmarks.
A new multi-agent learning method improves performance in complex games.
problem Performance gap between MAPG and value-based multi-agent approaches.
method Introduces value function decomposition into multi-agent actor-critic framework for off-policy learning.
result DOP significantly outperforms state-of-the-art multi-agent reinforcement learning algorithms.
A new method for RL with continuous actions improves stability and scalability.
problem Stability and scalability issues in existing RL methods.
method Soft policy gradient with entropy regularization, combined with double sampling for soft Bellman equation.
result Outperforms off-policy prior methods in continuous action RL tasks.
This paper improves sample efficiency for off-policy evaluation with preference data.
problem Improving sample efficiency for off-policy evaluation with preference data.
method Using a deep neural network to learn the value function and leveraging manifold structure.
result Established a provably efficient guarantee for off-policy evaluation with RLHF.
Proposes a value-based method for continuous control without an actor.
problem Computational infeasibility of evaluating Q-values in continuous action spaces.
method Structurally maximizable Q-functions, actor-free approach.
result Performance and sample efficiency comparable to actor-critic methods.
New method combines value function decomposition and policy gradients for cooperative multi-agent reinforcement learning.
problem Challenges in cooperative multi-agent reinforcement learning, especially credit assignment and large action spaces.
method Decomposed Soft Actor-Critic (mSAC) method with Q network architecture, discrete probabilistic policy, and counterfactual advantage function.
result Significantly outperforms policy-based approach COMA and achieves competitive results with SOTA value-based approach Qmix.
MODULE solves LfO problem with high sample efficiency and stability.
problem Sample inefficiency and instability in LfO.
method Modified DSAC with MODULE algorithm.
result Superior performance in MuJoCo environments.
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
problem Combining robust on-policy methods with efficient offline data for hybrid RL.
method Integrates off-policy training on offline data into on-policy NPG framework.
result Achieves state-of-the-art theoretical guarantees and maintains on-policy NPG guarantees.
SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.
problem Gradient ascent in complex Q-functions leads to suboptimal solutions.
method SAVO actor generates multiple action proposals and truncates poor local optima.
result SAVO actor finds optimal actions more frequently and outperforms other architectures.