Dreaming model enables robots to learn policies from images.
problem Learning to control robots based on images is challenging.
method Learning a future representation/scene regressor conditioned on robot actions.
result Dreaming model enables policy learning that transfers to real-world.
SDM Policy accelerates inference for robotic tasks while maintaining high action quality.
problem Prolonged inference times in diffusion-based policies for high-frequency control tasks.
method Two-stage optimization: score matching and distribution matching; dual-teacher mechanism.
result 6x inference speedup with state-of-the-art action quality.
A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a for…
Method learns multi-stage tasks from single video, overcoming challenges of raw pixel learning and insufficient demonstrations.
problem Learning multi-stage vision-based tasks from a single video of a human performing the task.
method Learn primitive behaviors from video demonstrations and dynamically compose them to perform multi-stage tasks.
result Demonstrated learning of various tasks on real robots using raw pixel inputs and minimal demonstrations.
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to replicate some of these abilities with a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically …
Robot learns to control itself without rewards.
problem Autonomous reinforcement learning without access to rewards.
method Distributional planning networks optimizing for an embedding space.
result Learned goal metrics enable autonomous reinforcement learning.
Robotic arm learns to manipulate a ball by choosing goals from learned experience.
problem Efficient discovery of skills for long-living autonomous agents without supervision.
method Intrinsically motivated goal exploration using learned goal spaces from deep representation learning.
result Recent results show applicability of learned goal spaces on real-world robotic tasks.
Few-shot goal learning for robot tasks from a few examples.
problem Learning objectives for robot tasks from limited examples.
method Meta-learning a classifier to recognize new goals from a few examples.
result Meta-learning approach can be applied to both reinforcement learning and visual planning.
New method corrects state distribution mismatch for off-policy policy optimization.
problem Mismatch between behavior and evaluation policy state distributions.
method Off-policy policy gradient with state distribution correction.
result Significantly improved policy quality in simulations.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
Paper improves off-policy evaluation by estimating behavior policy.
problem Evaluating policies with data from a different behavior policy.
method Importance sampling with an estimated behavior policy.
result Estimating behavior policy reduces mean squared error.
Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
Study shows accurate OPE depends on calibrated behaviour policy models.
problem Estimating a behaviour policy for OPE when true policy is unknown.
method Empirical studies comparing parametric vs non-parametric models.
result Simple non-parametric k-nearest neighbors model produces better calibrated behaviour policy estimates.
New framework studies policy learning problems under data scarcity.
problem Learning improving policies when data is insufficient.
method Developed a mathematical framework for policy learning problems.
result Reduced policy learning problems to simpler ones in sample complexity.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
Stabilizes policy optimization with off-policy data using divergence augmentation.
problem Premature convergence and instability in policy optimization with off-policy data.
method Incorporates Bregman divergence between behavior and current policies to ensure safe policy updates.
result Empirically shows better performance in data-scarce scenarios compared to other algorithms.
New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
New method improves off-policy policy evaluation by balancing policy values.
problem Accurate off-policy policy evaluation in reinforcement learning.
method Developed an MDP model with balanced representation to estimate both individual and average policy values.
result Substantially lower Mean Squared Error (MSE) in various benchmarks and a real-world HIV treatment simulation.
Normalizing flows policy improves trust region policy optimization.
problem Improving exploration and avoiding local optima in policy optimization.
method Constructing trust region with KL divergence constraints and using normalizing flows policy.
result Normalizing flows policy significantly improves policy optimization, especially on high-dimensional tasks.
POTEC tackles off-policy learning in large action spaces, improving effectiveness.
problem Existing OPL methods fail in large discrete action spaces due to bias or variance issues.
method Two-stage algorithm: cluster selection via policy-based approach, action selection via regression-based approach.
result POTEC provides substantial improvements in off-policy learning effectiveness, especially in large and structured action spaces.
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization …
A new theorem and algorithm solve off-policy policy gradient problems.
problem Solving the theoretical gap in off-policy policy gradient methods.
method Introduced an off-policy policy gradient theorem using emphatic weightings and developed the ACE algorithm.
result Demonstrated ACE finds the optimal solution in off-policy learning, unlike previous methods.
Protects proprietary policies from imitation learning by training adversarial policy ensembles.
problem Protecting policies from external observers cloning them.
method Introduces a reinforcement learning framework that trains an ensemble of near-optimal policies, making demonstrations useless for external observers.
result Demonstrates the existence of 'non-clonable' ensembles and provides a solution to the optimization problem.
Extends OPE to evaluate policies using diverse logging data.
problem Evaluate policies using log data from different policies.
method Develops an OPE method for various logging policies.
result Method's predictions converge to true performance as sample size increases.
PS framework selects best policy from library for CSO problems.
problem Policy selection in CSO with heterogeneous performance across covariate space.
method PS framework constructs library of candidate policies and learns a meta-policy to select the best one.
result PS consistently outperforms best single policy in heterogeneous CSO problems.
P3O merges on-policy and off-policy updates without extra hyper-parameters.
problem Combining on-policy and off-policy RL algorithms to reduce sample complexity.
method Interleaves off-policy updates with on-policy updates using effective sample size.
result P3O reduces sample complexity of state-of-the-art algorithms.
New method optimizes treatment policies to avoid winner's curse.
problem Winner's curse in treatment policy optimization.
method Inference-aware policy optimization.
result Optimizes for both estimated performance and downstream evaluation.
Entropy regularization improves policy optimization in reinforcement learning.
problem Improving policy optimization in reinforcement learning.
method Entropy regularization is introduced to soften the greedy policy towards a more diverse softmax policy, leading to a continuously parameterized algorithm that interpolates between policy gradient and Q-learning.
result An intermediate algorithm can improve performance in reinforcement learning.
PBVFs generalize across policies using learned value functions.
problem RL algorithms forget information about old policies when updating value functions to track the learned policy.
method Introduce Parameter-Based Value Functions (PBVFs) that include policy parameters in their inputs, enabling them to generalize across different policies.
result PBVFs enable zero-shot learning of new policies that outperform any policy seen during training.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Estimates policy value from off-policy data in contextual bandits.
problem Estimating policy value from limited off-policy data in contextual bandits.
method Empirical likelihood techniques for optimization.
result Improves over previous methods in finite sample regimes.
Policy gradient aims to maximize expected return using gradient ascent.
problem Finding a policy that maximizes expected return in a given class of policies.
method Gradient ascent applied to a differentiable model of the policy, estimating the gradient of expected return.
result Policy gradient methods require on-policy data for gradient estimation, limiting sample efficiency.
Improves on-policy RL by reusing data from multiple policies.
problem Lack of reuse of data from previous policies in on-policy RL.
method Adapts replay buffer concept to combine on- and off-policy learning.
result Method outperforms state-of-the-art on-policy RL algorithms.
Optimizes Thompson sampling policies using policy gradient methods.
problem Improving Thompson sampling in bandit problems.
method Applies policy gradient algorithms to optimize Thompson sampling policies.
result Direct policy search on Thompson sampling improves performance.
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
Paper introduces a new policy optimization method using importance sampling.
problem Stable and low variance policy learning with small policy updates.
method Derives an alternative objective using importance sampling and introduces an approximation to balance bias and variance.
result The new algorithm improves on-policy policy optimization on continuous control benchmarks.
This paper introduces a new method to evaluate multiple policies simultaneously.
problem Estimating the value of many policies for a single set of states.
method Developed a scalable, differentiable fingerprinting mechanism to represent complex policies.
result The method can produce policies that outperform those that generated the training data, in zero-shot manner.
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
Implicit Policy simplifies complex reinforcement learning policies.
problem Complex action distributions in reinforcement learning.
method Rich policy class with entropy regularization.
result Entropy regularization with rich policy class achieves desirable properties.
This work uses MMD to find diverse policies in reinforcement learning.
problem Finding multiple near-optimal policies for a task.
method Formalizes policy difference as trajectory distribution discrepancy, uses MMD for optimization.
result Derives gradient-based optimization for diverse policy identification.
Paper develops a new multi-agent reinforcement learning algorithm.
problem Improving policies in a network of communicating agents.
method Develops a multi-agent off-policy actor-critic algorithm using emphatic temporal difference learning.
result Proves convergence of the algorithm under linear function approximation.
New method reduces state distribution mismatch in off-policy RL.
problem State distribution mismatch in off-policy RL algorithms.
method Develops a novel constrained off-policy gradient objective to minimize state distribution shift.
result Minimizing state distribution shift improves performance in off-policy RL algorithms.
This work analyzes the gap between off-policy and on-policy policy gradient methods and provides conditions to reduce this gap.
problem The gap between off-policy and on-policy policy gradient methods and conditions to reduce it.
method Theoretical analysis and empirical evidence of conditions to reduce the on-off gap.
result Conditions to reduce the on-off gap between off-policy and on-policy policy gradient methods.