arXiv research

Variational Proximal Policy Optimization improves reinforcement learning from human feedback.

Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies

Proposes a Riemannian optimization for policy improvement in MDPs.

New algorithm stabilizes RL policy learning through divergence regularization.

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

Improves reinforcement learning stability and efficiency.

PPOS improves PPO by smoothing the surrogate objective function.

We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…

Improves RL algorithms with two techniques.

Revisits PPO design choices, exposing failure modes and proposing alternatives.

In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (…

Policy-gradient method controls multiple non-cohesive targets.

Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which leads to slow progress and may make the algorithm prone to getting stuck in local optima. Drawing insp…

Paper proposes a new method to optimize robot body structure and control policy.

OPPO optimizes policy learning with provable efficiency.

Proximal policy optimization(PPO) has been proposed as a first-order optimization method for reinforcement learning. We should notice that an exterior penalty method is used in it. Often, the minimizers of the exterior penalty functions approach feasibility only in the limits as the penalty parameter grows increasingly…

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…

Study shows code-level optimizations significantly impact deep RL algorithms.

Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restr…

As the most successful variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely applied across various domains with several advantages: efficient data utilization, easy implementation, and good parallelism. In this paper, a first-order gradient reinforcemen…

Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves da…

PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.

This research introduces an autonomous robot navigation method using reinforcement learning.

Study shows tractable generalization in RL is impossible but possible with Strong Proximity.

This paper uses deep reinforcement learning to optimize stock portfolios considering transaction costs and risks.

RPO uses past and future state-action info for better policy optimization.

Paper uses DRL to optimize portfolios, balancing risk and return.

Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this pa…

Develops ODRPO to improve RL algorithms with better performance and stability.

Adapts GRPO for off-policy RL, improving reward.

AI algorithms outperform traditional trading methods in stock markets.

A drone catches another agile drone using competitive reinforcement learning.

TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.

Photonic quantum reinforcement learning for control problems.

Develops minibatch stochastic proximal gradient for large-scale learning models.

The paper tackles counterfactual learning for stochastic policies with continuous actions.

Recent policy optimization approaches (Schulman et al., 2015a; 2017) have achieved substantial empirical successes by constructing new proxy optimization objectives. These proxy objectives allow stable and low variance policy learning, but require small policy updates to ensure that the proxy objective remains an accur…

This paper compares eight DRL algorithms for dynamic hedging.

MAP-Elites generates diverse trading strategies for improved execution performance.

Machine learning improves Buchberger's algorithm for polynomial systems.

This work extends ME-RL using diffusion models to sample optimal policies.

WAPPO optimizes feature distributions for better visual transfer in RL.

PRL improves off-policy evaluation in partially observed MDPs.

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…

In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course. Top participants were invited to describe their algorithms. In this work, we present eight solutions that used deep reinforcement lea…

This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is t…

Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous exploration methods either rely on complex structure to estimate the novelty of states, or incur sens…