DPG makes reinforcement learning safer with human advice.
problem Unsafe reinforcement learning in shared environments.
method Extends Policy Gradient to incorporate human directives.
result DPG learns faster and more safely than reward-based methods.
Direct policy gradients optimize policies in discrete action spaces using sampling.
problem Optimizing policies in discrete action spaces with direct methods.
method Combining direct optimization and A ⋆ ^\star ⋆ sampling for policy gradient approximation. result DirPG algorithms can incorporate domain knowledge and have higher probability of sampling informative gradients.
Direct and indirect RL methods classified and compared.
problem Classifying RL methods for sequential decision making.
method Direct RL solves optimal policy directly, indirect RL solves Bellman equation.
result Direct and indirect RL methods are equivalent and can be unified.
Paper tackles efficient policy gradient estimation from off-policy data.
problem Estimating policy gradients from off-policy data is challenging and inefficient.
method Derives asymptotic lower bounds, proposes a meta-algorithm with 3-way robustness, and establishes convergence guarantees.
result Meta-algorithm achieves the lower bound on mean-squared error without parametric assumptions.
Policy gradient methods do not optimize the discounted objective, leading to suboptimal results.
problem Understanding the true optimization objective of policy gradient methods.
method Analyzing the update direction of policy gradient methods and proving it is not the gradient of any function.
result Policy gradient methods do not optimize the discounted objective, leading to suboptimal results.
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
DG improves policy gradients by weighting actions with a sigmoid of advantage and surprisal.
problem Pathologies in standard policy gradients, leading to poor updates and over-allocation of gradient budget.
method Introduces Delightful Policy Gradient (DG) that gates each term with a sigmoid of advantage and surprisal.
result DG provably improves directional accuracy in a single context and shifts the expected gradient closer to the oracle across multiple contexts.
This work improves Hindsight Learning for goal-directed tasks in reinforcement learning.
problem Sparse reward problems in reinforcement learning, especially goal-directed tasks.
method Improves Hindsight Experience Replay and Hindsight Policy Gradients for robotic tasks.
result Improved learning in Hindsight Experience Replay and Hindsight Policy Gradients.
Optimizes Thompson sampling policies using policy gradient methods.
problem Improving Thompson sampling in bandit problems.
method Applies policy gradient algorithms to optimize Thompson sampling policies.
result Direct policy search on Thompson sampling improves performance.
Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over R d \mathbb{R}^d R d and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are natu…
Integrates multiple feedback channels into policy learning for reinforcement learning.
problem Combining multiple types of feedback into policy gradient algorithms.
method Lagrangian relaxation to satisfy constraints using gradient descent while maximizing rewards.
result Constraints are respected and can accelerate learning in reinforcement learning tasks.
PC-PG balances exploration and exploitation in reinforcement learning.
problem Local policy gradient methods struggle with exploration.
method PC-PG uses an ensemble of learned policies (policy cover) to balance exploration and exploitation.
result PC-PG provides strong theoretical guarantees and empirical validation.
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.
problem Continuous state-action reinforcement learning problems.
method Introduced an alternative policy gradient theorem using weak derivatives.
result The new approach yields algorithms that converge almost surely to stationary points of the value function.
This paper introduces a new method to evaluate multiple policies simultaneously.
problem Estimating the value of many policies for a single set of states.
method Developed a scalable, differentiable fingerprinting mechanism to represent complex policies.
result The method can produce policies that outperform those that generated the training data, in zero-shot manner.
Paper develops methods to optimize policies directly from human feedback without reward inference.
problem Challenges in RLHF, including reward model overfitting and distribution shift.
method Develops two algorithms for RLHF without reward inference, using zeroth-order gradient approximators.
result Establishes polynomial convergence rates and outperforms existing methods in numerical experiments.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m − 1 / 2 m^{-1/2} m − 1/2 . This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
problem The bias-variance tradeoff in estimating and using policy gradients for deep reinforcement learning.
method Introduces elite policy gradients and a two-step merging method to balance bias-variance tradeoffs.
result Two-step merging outperforms interpolation merging and state-of-the-art algorithms on benchmark control tasks.
New method for RL with general utilities using variational policy gradient.
problem Optimizing policies with general concave utility functions in RL.
method Derives Variational Policy Gradient Theorem, develops variational Monte Carlo gradient estimation algorithm.
result Global convergence to optimal policy for general objectives, exponential convergence under strong convexity.
Study policy gradient for large-agent mean-field control and game in continuous time.
problem Optimal policy learning for large number of agents in continuous-time mean-field systems.
method Policy gradient method applied to linear-quadratic mean-field control and game models.
result Policy gradient converges to optimal solution at a linear rate for both mean-field control and game.
This paper uses deep reinforcement learning to optimize stock portfolios considering transaction costs and risks.
problem Optimizing stock portfolios with transaction costs and risks.
method Formulated stock portfolio optimization as a reinforcement learning problem, applied DDPG, GDPG, and PPO algorithms, and used Wavelet Transform.
result DDPG and GDPG algorithms outperformed PPO in continuous action space.
We study a reinforcement learning setting, where the state transition function is a convex combination of a stochastic continuous function and a deterministic function. Such a setting generalizes the widely-studied stochastic state transition setting, namely the setting of deterministic policy gradient (DPG). We firstl…
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
PIPPS solves deep learning's exploding gradient problem by reparameterization gradients.
problem Exploding gradients in deep learning and model-based RL.
method Develops PIPPS framework, a flexible policy search method robust to chaos-like gradients.
result PIPPS improves over reparameterization gradients by up to 10^6 times.
The paper analyzes RLVR's training dynamics, proving convergence depends on aligning update direction with Gradient Gap.
problem Understanding why RLVR works and its limitations.
method Analysis of RLVR's training process at trajectory and token levels, introducing Gradient Gap.
result Convergence depends on aligning update direction with Gradient Gap, with a sharp step-size threshold.
Survey of theoretical foundations for policy optimization in control.
problem Understanding the theoretical properties of gradient-based methods in control and reinforcement learning.
method Interdisciplinary review of optimization landscape, convergence, and sample complexity for various control problems.
result Recent theoretical results on stability and robustness in learning-based control.
SAUNA filters out noisy samples to boost RL performance.
problem Improving RL performance by filtering out non-informative samples.
method SAUNA selects samples based on the fraction of variance explained by the value function, rejecting non-informative transitions.
result SAUNA significantly improves RL performance on benchmark problems.
ZSPO optimizes RL from unknown link functions using human feedback.
problem Designing RLHF algorithms for unknown link functions.
method Zero-order policy optimization with human preference feedback.
result ZSPO converges to a stationary policy with a polynomial rate.
A new ES method improves reinforcement learning speed and accuracy.
problem Slow convergence and local maxima in reinforcement learning.
method Directional Gaussian Smoothing Evolution Strategy (DGS-ES)
result DGS-ES accelerates RL training with high accuracy and nonlocal search direction.
Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they…
New algorithm converges to optimal filter for predicting linear dynamical systems.
problem Direct policy search for optimal dynamic filters in partially observable systems.
method Regularizer enforcing informativity over filter states.
result Gradient descent converges to globally optimal solution at rate O(1/T).
QAM uses adjoint matching to optimize continuous-action RL policies efficiently.
problem Efficient optimization of expressive diffusion or flow-matching policies with respect to a Q-function.
method QAM leverages adjoint matching to bypass the numerical instability of backpropagation through multi-step denoising processes.
result QAM consistently outperforms prior approaches on hard, sparse reward tasks in offline and offline-to-online RL.
In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…
New PG losses improve decision optimization in misspecified models.
problem Improving decision optimization in models that are not perfectly specified.
method Introducing Perturbation Gradient (PG) losses to connect decision loss with directional derivatives and optimizing using gradient techniques.
result PG losses yield best-in-class policies asymptotically, even in misspecified settings.
The paper presents a model-free method for stabilizing unknown control systems.
problem Stabilizing unknown control systems in engineering.
method Solving discounted LQR problems with increasing discount factors.
result The method efficiently recovers a stabilizing controller for linear and smooth nonlinear systems.
ALIAS uses RL to learn DAGs without acyclicity constraints.
problem Efficiently learning DAGs from observational data without acyclicity constraints.
method ALIAS employs RL to generate DAGs in a single step with optimal complexity, bypassing acyclicity constraints.
result ALIAS outperforms state-of-the-art methods in causal discovery.
The paper optimizes regret using covariance between costs and decisions.
problem Optimizing expected regret in decision-making problems.
method Developed derivative theory of covariance regret functional, derived Gâteaux derivative, and extended to constrained optimization.
result Gradient of covariance regret is the cost covariance matrix, with implications for portfolio optimization.
Risk-sensitive RL aims to optimize performance while considering risk constraints or measures.
problem Optimizing expected value alone may not be sufficient; incorporating risk measures is necessary.
method Policy gradient search is used to solve risk-sensitive RL problems, considering various risk measures.
result Policy gradient methods can be adapted for risk-sensitive RL, addressing challenges and future directions.
Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…
Policy gradient aims to maximize expected return using gradient ascent.
problem Finding a policy that maximizes expected return in a given class of policies.
method Gradient ascent applied to a differentiable model of the policy, estimating the gradient of expected return.
result Policy gradient methods require on-policy data for gradient estimation, limiting sample efficiency.
OPPO optimizes policy learning with provable efficiency.
problem Designing efficient policy optimization algorithms that incorporate exploration.
method Proposes an Optimistic Proximal Policy Optimization (OPPO) algorithm.
result Achieves i l d e O ( d 2 H 3 T ) ilde{O}(\sqrt{d^2 H^3 T} ) i l d e O ( d 2 H 3 T ) regret in episodic MDPs. Algorithm optimizes system design and control for better rewards.
problem Optimizing system design and control for maximum rewards.
method Deep reinforcement learning combining policy gradient and model-based optimization.
result DEPS algorithm outperforms state-of-the-art methods in various environments.
ARM policy gradient reduces variance for binary actions.
problem High variance in policy gradients for binary actions.
method Augment-Reinforce-Merge (ARM) policy gradient estimator.
result ARM estimator achieves significant variance reduction and faster convergence.
Unified approach to Merton's portfolio problem using Pontryagin's principles.
problem Optimizing consumption and investment strategies in financial portfolios.
method PG-DPO framework combining neural networks with Pontryagin's maximum principle.
result Locally optimal policies closely tied to classical stochastic control.
A new theorem and algorithm solve off-policy policy gradient problems.
problem Solving the theoretical gap in off-policy policy gradient methods.
method Introduced an off-policy policy gradient theorem using emphatic weightings and developed the ACE algorithm.
result Demonstrated ACE finds the optimal solution in off-policy learning, unlike previous methods.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
New method combines value function decomposition and policy gradients for cooperative multi-agent reinforcement learning.
problem Challenges in cooperative multi-agent reinforcement learning, especially credit assignment and large action spaces.
method Decomposed Soft Actor-Critic (mSAC) method with Q network architecture, discrete probabilistic policy, and counterfactual advantage function.
result Significantly outperforms policy-based approach COMA and achieves competitive results with SOTA value-based approach Qmix.