The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
Proposes CoPO, a new policy optimization method for competitive games.
problem Designing efficient optimization methods for competitive Markov decision processes.
method Competitive policy optimization (CoPO) approach that exploits game-theoretic nature of competitive games.
result Stable optimization, convergence to sophisticated strategies, and higher scores compared to baseline methods.
RPO uses past and future state-action info for better policy optimization.
problem Sample inefficiency in on-policy reinforcement learning methods.
method Reflective Policy Optimization (RPO) integrates past and future state-action info for policy improvement.
result RPO improves policy performance and contracts the solution space, leading to faster convergence.
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
New method optimizes treatment policies to avoid winner's curse.
problem Winner's curse in treatment policy optimization.
method Inference-aware policy optimization.
result Optimizes for both estimated performance and downstream evaluation.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
Recent policy optimization approaches (Schulman et al., 2015a; 2017) have achieved substantial empirical successes by constructing new proxy optimization objectives. These proxy objectives allow stable and low variance policy learning, but require small policy updates to ensure that the proxy objective remains an accur…
Stabilizes policy optimization with off-policy data using divergence augmentation.
problem Premature convergence and instability in policy optimization with off-policy data.
method Incorporates Bregman divergence between behavior and current policies to ensure safe policy updates.
result Empirically shows better performance in data-scarce scenarios compared to other algorithms.
Accelerates policy optimization in RL with optimistic and adaptive updates.
problem Improving policy optimization methods in reinforcement learning.
method Integrates foresight into policy improvement step via optimistic and adaptive updates.
result Designs an optimistic policy gradient algorithm, adaptive via meta-gradient learning.
New method optimizes policies in non-stationary environments.
problem Optimizing policies in non-stationary, context-dependent environments.
method Two-phase approach: offline learning and online adaptation.
result Our method outperforms existing approaches in both synthetic and real-world datasets.
Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
Proposes a Riemannian optimization for policy improvement in MDPs.
problem Optimizing policy functions in Markov decision processes (MDPs).
method Riemannian proximal optimization algorithm with Gaussian mixture model (GMM).
result Guaranteed convergence and efficacy demonstrated in preliminary experiments.
Develops an anytime-valid framework for optimal policy identification from logged contextual bandit data.
problem Selecting the optimal policy from a candidate policy class while monitoring evidence continuously.
method Constructs a time-indexed set that retains the true optimal policy set uniformly over time.
result The procedure allows the analyst to monitor policy values, eliminate clearly suboptimal policies, and stop at data-dependent times without invalidating inference.
Entropy regularization improves policy optimization in reinforcement learning.
problem Improving policy optimization in reinforcement learning.
method Entropy regularization is introduced to soften the greedy policy towards a more diverse softmax policy, leading to a continuously parameterized algorithm that interpolates between policy gradient and Q-learning.
result An intermediate algorithm can improve performance in reinforcement learning.
PS framework selects best policy from library for CSO problems.
problem Policy selection in CSO with heterogeneous performance across covariate space.
method PS framework constructs library of candidate policies and learns a meta-policy to select the best one.
result PS consistently outperforms best single policy in heterogeneous CSO problems.
Paper addresses offline policy evaluation in RL, achieving near-optimal bounds for various policy classes.
problem Evaluate all policies in a class simultaneously for offline RL.
method Uniform convergence in OPE for various policy classes, achieving optimal episode complexity.
result Achieves optimal episode complexity of O(H^3/d_mε^2) for identifying ε-optimal policies.
Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate. Though often achieving encouraging empirical success, its underlying mathematical principle on {\em policy-distri…
Optimizes Thompson sampling policies using policy gradient methods.
problem Improving Thompson sampling in bandit problems.
method Applies policy gradient algorithms to optimize Thompson sampling policies.
result Direct policy search on Thompson sampling improves performance.
Presents SPEED, an algorithm for optimal policy evaluation in linear bandits with heteroscedastic noise.
problem Optimal data collection for policy evaluation in linear bandits with heteroscedastic reward noise.
method Formulated an optimal design for weighted least squares estimates, derived the optimal sample allocation, introduced SPEED algorithm, and derived regret bounds.
result SPEED leads to policy evaluation with MSE comparable to oracle strategy and significantly lower than random policy execution.
A new method removes policy optimization in adversarial imitation learning.
problem Adversarial imitation learning's delicate alternated optimization.
method Explicitly condition discriminator on two policies, solving generator's optimization problem directly.
result Simpler approach competitive to prevalent methods.
Policy gradient converges to globally optimal policy in nearly linear-quadratic systems.
problem Finding optimal policies in nonlinear control systems with partial information.
method Policy gradient algorithm designed for nearly linear-quadratic regulators with small Lipschitz nonlinear components.
result Policy gradient algorithm converges to globally optimal policy with linear rate.
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
New algorithm finds near-optimal policies efficiently in zero-sum games.
problem Lack of provable efficiency guarantees for policy optimization in zero-sum games.
method Policy optimization algorithm with function approximation.
result Proves efficient convergence to near-optimal policies with polynomial samples and iterations.
A usual reinsurance policy for insurance companies admits one or two layers of the payment deductions. Under optimal criterion of minimizing the conditional tail expectation (CTE) risk measure of the insurer's total risk, this article generalized an optimal stop-loss reinsurance policy to an optimal multi-layer reinsur…
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
problem Some policy optimization algorithms do not have batch size-invariance, leading to inefficiencies.
method We decouple the proximal policy from the behavior policy to achieve batch size-invariance.
result Our approach makes policy optimization algorithms more efficient and allows them to use stale data more effectively.
Taylor expansions improve reinforcement learning policies.
problem Improving reinforcement learning policy optimization.
method Taylor expansion policy optimization.
result Taylor expansions enhance performance of distributed algorithms.
Improves reinforcement learning stability and efficiency.
problem Combining stability and efficiency in reinforcement learning.
method Combines on-policy stability with off-policy sample reuse.
result Demonstrates improved performance in both theory and practice.
Lo-Hp decouples weight generation into local and global policies to improve flexibility and efficiency.
problem Over-coupling and long-horizon issues in current optimization methods.
method Hybrid-Policy Sub-Trajectory Balance objective.
result Learning local optimization policies addresses long-horizon issues and enhances global weight generation.
Optimizes antenna tilt for better QoS in cellular networks.
problem Hard to learn optimal antenna tilt policies in real networks due to risk and simulation gap.
method Uses off-policy Contextual Multi-Armed-Bandit (CMAB) techniques to learn from existing data.
result Trained policies show consistent improvements over existing logging policies.
Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbi…
New findings show optimization is crucial for OPL in large action spaces.
problem Challenges in optimizing policies for large action spaces in offline contextual bandits.
method Weighed log-likelihood objectives and estimator-aware policy parametrization.
result Simple weighted log-likelihood objectives enjoy better optimization properties and recover competitive policies.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
New policy optimizes risk and optimality in stochastic bandits.
problem Optimizing risk in stochastic bandits with heavy-tailed risk.
method Designing policies with worst-case optimality for expected regret and light-tailed risk distribution.
result Achieves worst-case optimality for expected regret and light-tailed risk distribution.
This work extends ME-RL using diffusion models to sample optimal policies.
problem Sampling from the optimal policy trajectory distribution in ME-RL.
method Introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs) to minimize reverse KL divergence.
result DA-MDPs enable seamless integration into various ME-RL methods and outperform baselines.
New method for estimating value of optimal policies in uncertain scenarios.
problem Inference for optimal policies when they are non-unique or nearly deterministic.
method Semiparametric efficiency bound, uniformly weighted estimator, NSAVE method.
result Proposes NSAVE method for robust inference in uncertain optimal policies.
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.
problem Policy Optimization in continuous control tasks.
method RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for regret minimization in PO.
result Achieving constant regret under certain circumstances in PO with mediator feedback.
Quadratic hedging of option payoffs generates the variance optimal martingale measure. When an option features an exercise policy and its cash flows are hedged according to this approach, it may be tempting to optimize such a policy under this measure. Because the variance optimal martingale measure may not be an equiv…
PPO algorithm converges to global optimality in multi-agent reinforcement learning.
problem Designing statistical guarantees for policy optimization methods in multi-agent reinforcement learning.
method Leveraging a multi-agent performance difference lemma, a localized action value function is used as a descent direction for each local policy, leading to a multi-agent PPO algorithm.
result The multi-agent PPO algorithm converges to the globally optimal policy at a sublinear rate under standard regularity conditions.
This paper investigates methods for estimating the optimal stochastic control policy for a Markov Decision Process with unknown transition dynamics and an unknown reward function. This form of model-free reinforcement learning comprises many real world systems such as playing video games, simulated control tasks, and r…
Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.
problem Optimizing softmax policies with neural networks in the mean-field regime.
method Modeling neural networks as Wasserstein gradient flows and proving global optimality of fixed points.
result Global optimality of softmax policy gradient in wide single hidden layer neural networks with entropy regularization.
Variational Proximal Policy Optimization improves reinforcement learning from human feedback.
problem Policy mode collapse and brittle exploration loops in reinforcement learning.
method Particle-based variational inference framework with Mixture-of-Experts architecture.
result Significant improvements in complex reasoning benchmarks.
Paper improves policy updates in reinforcement learning to speed up learning.
problem Slow learning and unlearning in policy optimization.
method Introduces a novel gradient update and a modified policy update.
result Proves modified policy update converges to global optimality.
Survey explores geometric aspects of policy optimization in control systems.
problem Understanding the geometric relationships between control design and optimization.
method Geometric perspective on policy optimization, focusing on parameterization and topology.
result Implications of policy geometry on stability and performance of local search algorithms.