Improves reinforcement learning stability and efficiency.
problem Combining stability and efficiency in reinforcement learning.
method Combines on-policy stability with off-policy sample reuse.
result Demonstrates improved performance in both theory and practice.
A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or Q-function may fail to improve performance---or worse, actually cause the policy performance …
New RL algorithms improve control tasks with data reuse.
problem Real-world control requires performance guarantees and data efficiency.
method Generalized Policy Improvement combining on-policy guarantees and sample reuse.
result Extensive experimental analysis shows benefits of new algorithms.
This paper improves reinforcement learning policies in a scalable way.
problem Ensuring monotonic policy improvement in entropy-regularized RL.
method Derives an entropy-aware lower bound and proposes a novel RL algorithm.
result Demonstrates effectiveness in continuous-state tasks using a linear function approximator.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraints, normalizing flows policy generates samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps av…
Proposes a Riemannian optimization for policy improvement in MDPs.
problem Optimizing policy functions in Markov decision processes (MDPs).
method Riemannian proximal optimization algorithm with Gaussian mixture model (GMM).
result Guaranteed convergence and efficacy demonstrated in preliminary experiments.
New method improves policies by combining Markov and non-Markov strategies.
problem Improving policies in reinforcement learning.
method Geometric Policy Composition (GPI) using a geometric horizon model (GHM).
result GPI can improve non-Markov policies by composing GHMs of base Markov policies.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Improves BC policies by generating new plausible trajectories.
problem Sub-optimal data quality in BC leads to poor policy performance.
method Trajectory Stitching (TS) generates new plausible transitions.
result TS significantly improves behavioural policies over original data.
Framework improves policy generalizability under biased training data.
problem Learning policies that generalize to a target population from biased training data.
method Characterizes sample selection bias using a selection variable, optimizes minimax value over uncertainty set, derives efficient algorithm.
result Policies generalize to target population, outperform standard methods.
Taylor expansions improve reinforcement learning policies.
problem Improving reinforcement learning policy optimization.
method Taylor expansion policy optimization.
result Taylor expansions enhance performance of distributed algorithms.
New framework studies policy learning problems under data scarcity.
problem Learning improving policies when data is insufficient.
method Developed a mathematical framework for policy learning problems.
result Reduced policy learning problems to simpler ones in sample complexity.
New method improves reinforcement learning generalization.
problem Few environments lead to poor generalization in reinforcement learning.
method Integrates sequential structure into representation learning, using a policy similarity metric (PSM) and contrastive embeddings (PSEs).
result PSEs improve generalization across various benchmarks.
We propose a policy improvement algorithm for Reinforcement Learning (RL) which is called Rerouted Behavior Improvement (RBI). RBI is designed to take into account the evaluation errors of the Q-function. Such errors are common in RL when learning the Q-value from finite past experience data. Greedy policies or even …
Paper analyzes model usage in policy optimization, improving sample efficiency and performance.
problem Balancing ease of data generation with model bias in reinforcement learning.
method Formulated and analyzed model-based reinforcement learning algorithm with empirical model generalization.
result Simple model-based approach outperforms existing methods in sample efficiency and asymptotic performance.
Improves RL generalization by minimizing adversarial risk.
problem Overfitting to training environments and poor generalization to unseen scenarios.
method Introduces minimax formulation and distributional framework to RL.
result Trained policy shows improved generalization to different environments.
Improves policies with high certainty, even in small samples.
problem Ensuring new policies are better than the baseline with high probability.
method Leverages powerful safety tests and multiple testing for threshold policies.
result Controls the rate of adopting a worse policy to pre-specified error level.
New RL method learns value function for many policies using few key states.
problem Evaluate and improve policies in continuous control problems.
method Combines actor-critic architecture and policy embedding to learn a single value function for many policies.
result Value function minimizes prediction error by learning a small set of 'probing states' and their impact on policies' returns.
New algorithm for sequential off-policy learning improves performance over batch methods.
problem Training policies from logged interaction data in a sequential setting.
method Combines Logarithmic Smoothing with online PAC-Bayesian tools.
result Improves performance and accelerates convergence in sequential off-policy learning.
To ensure stability of learning, state-of-the-art generalized policy iteration algorithms augment the policy improvement step with a trust region constraint bounding the information loss. The size of the trust region is commonly determined by the Kullback-Leibler (KL) divergence, which not only captures the notion of d…
Paper improves safe policy improvement with estimated baseline policy.
problem Unreliable batch Reinforcement Learning algorithms in real-world applications.
method Apply SPIBB algorithms with an estimated baseline policy.
result Safe policy improvement guarantees over true baseline without direct access.
Enhances PMD with lookahead to improve RL performance.
problem Improving RL performance with greedy policies over 1-step.
method Integrates multi-step greedy policies into PMD with lookahead.
result Shows faster convergence rate for h-PMD. The paper develops a safe policy gradient algorithm for reinforcement learning.
problem Safety issues in reinforcement learning for real-world control tasks.
method Stochastic optimization perspective, meta-parameter schedules, adaptive selection of step size and batch size.
result Monotonic improvement guarantees for a wide class of parametric policies.
This work improves policy evaluation and selection using logarithmic smoothing for pessimistic off-policy estimation.
problem Offline evaluation and selection of policies from past data.
method Develops novel concentration bounds and a logarithmically smoothed estimator (LS) for improved policy selection and learning.
result The logarithmically smoothed estimator (LS) provides tighter bounds and better policy selection and learning.
DBQPG improves policy gradient estimation with fewer samples.
problem Accurate policy gradient estimation with limited samples.
method Deep Bayesian Quadrature Policy Gradient (DBQPG).
result DBQPG provides more accurate and less variable gradient estimates.
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization …
Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
SPIBB improves safe policy improvement with a softer baseline approach.
problem Improving policies safely in reinforcement learning.
method Baseline Bootstrapping algorithm (SPIBB) that allows policy search over a wider set of policies, controlling policy change according to local model uncertainty.
result Significant improvement in safe policy improvement over existing methods.
Improved off-policy selection and learning in contextual bandits with better guarantees.
problem Selecting or training a reward-maximizing policy using data from a fixed behavior policy.
method A betting-based confidence bound applied to an inverse propensity weight sequence for off-policy selection, and a freezing condition for off-policy learning.
result The proposed methods achieve significantly improved guarantees over prior work, especially in small-data regimes.
Unified reinforcement learning objective improves policy performance prediction.
problem Improving policy performance prediction in reinforcement learning.
method Proposed a counterfactual objective and an emphatic approach to compute unbiased policy gradients.
result Geoff-PAC algorithm demonstrates empirical success in deep RL benchmarks.
Improves off-policy evaluation weights for balanced state-action pair distribution.
problem Imbalance in importance sampling weights for off-policy evaluation of contextual bandits.
method Balanced Off-Policy Evaluation (B-OPE) method that minimizes imbalance to desired counterfactual distribution of state-action pairs.
result Experimental evidence shows B-OPE improves offline policy evaluation in both discrete and continuous action spaces.
VRER selectively reuses samples to improve policy optimization in complex systems.
problem Lack of effective reuse of historical samples in reinforcement learning.
method Variance reduction based experience replay (VRER) framework.
result VRER accelerates policy optimization and enhances performance.
Improved RCPs for MABs using normalized weight functions.
problem Slow convergence and inferior rewards in RCPs for MABs.
method Generalized marginalization of rewards using normalized weight functions.
result Improved RCPs become competitive with classic methods.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
This work explains why online imitation learning improves faster than theory predicts.
problem Online imitation learning's empirical policy improvement speed exceeds theoretical predictions.
method The authors analyze online imitation learning with a convex, smooth, and non-negative loss function, proving policy improvement in expectation and high probability.
result Adopting a sufficiently expressive policy class in online IL increases both policy improvement speed and performance bias.
Double descent found in DRL, improving generalization with model capacity.
problem Generalization in over-parameterized DRL models.
method Actor-Critic framework, Policy Entropy metric.
result Policy Entropy significantly reduces as model capacity increases, indicating improved generalization.
Extends OPE to evaluate policies using diverse logging data.
problem Evaluate policies using log data from different policies.
method Develops an OPE method for various logging policies.
result Method's predictions converge to true performance as sample size increases.
TBQ(σ) improves trace utilization in off-policy reinforcement learning.
problem Efficiency of trace utilization under greedy target policies.
method Introduces TBQ(σ) that unifies tree-backup and Naive Q(λ) with a new parameter σ.
result TBQ(σ) improves efficiency in trace utilization, accelerating learning and performance.
Adversarial AutoAugment improves image classification with less computation.
problem Improving deep neural network generalization through data augmentation.
method Adversarial method to optimize augmentation policies without retraining the target network.
result Significant performance improvements on ImageNet and CIFAR-10.
Agents compose pre-trained policies for complex tasks, improving zero-shot performance.
problem Challenges in long-horizon predictions and estimating visitation distributions induced by policy sequences.
method Learn predictive jumpy world models of multi-step dynamics, enhancing predictions with a consistency objective.
result Compositional planning with jumpy world models yields, on average, a 200% relative improvement over primitive actions on long-horizon tasks.
Active-GRPO improves molecular optimization by actively deciding when to imitate or self-improve.
problem Training robust and efficient molecular optimization models with large language models.
method Active-GRPO combines imitation and reinforcement learning, upgrading references and policies dynamically.
result Improves molecular optimization performance, achieving statistically significant gains.
New GFlowNet training framework using policy gradients for combinatorial object generation.
problem Training efficiency and robustness in GFlowNet models.
method Policy-dependent rewards and coupled training strategy for forward and backward policies.
result Advanced RL perspectives for robust gradient estimation improve GFlowNet performance.
The paper improves DR estimators for off-policy evaluation in contextual bandits.
problem Challenges in evaluating policies with adaptively collected data.
method Adaptive weighting to control variance in DR estimators.
result Improved t-statistic based on the estimator is asymptotically normal.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
A new framework improves reinforcement learning algorithms with policy guarantees.
problem Designing efficient and stable reinforcement learning algorithms.
method A general framework (FMA-PG) based on functional mirror ascent that constructs surrogate functions enabling policy improvement guarantees.
result The proposed framework enables policy improvement guarantees that hold regardless of policy parameterization, and recovers important heuristics.
VERAFI improves financial AI by verifying calculations and compliance.
problem Financial AI systems generate errors and violations during reasoning.
method VERAFI combines dense retrieval, reranking, and automated reasoning policies.
result VERAFI achieves 94.7% factual correctness, 81% relative improvement.
Paper tackles reinforcement learning generalization through invariant policy optimization.
problem Learning policies that generalize beyond training domains.
method Invariant policy optimization principle and novel learning algorithm IPO.
result Significant improvements in generalization performance on unseen domains.