Model-based reinforcement learning algorithms tend to achieve higher sample efficiency than model-free methods. However, due to the inevitable errors of learned models, model-based methods struggle to achieve the same asymptotic performance as model-free methods. In this paper, We propose a Policy Optimization method w…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Model-based Policy Learning”
MoMA improves model-based RL by using unrestricted policy classes.
problem Limited sample efficiency and generalizability in model-based offline RL.
method Model-based mirror ascent algorithm with general function approximations.
result Theoretical guarantees and practical implementation of MoMA.
Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introd…
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
VMBPO optimizes model and policy jointly using variational lower-bound.
problem Data efficiency in RL with biased simulated data.
method Formulate variational objective function, use EM, iteratively improve model and policy.
result VMBPO is more sample-efficient and robust than model-free algorithms.
We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, -functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal -func…
Unified framework for policy improvement in RL with benefits in data efficiency and computation.
problem Improving data efficiency and computation in reinforcement learning for continuous control.
method Local, regularized policy improvement with tree search for continuous action spaces.
result Improves data efficiency and reduces wall-clock time in high-dimensional domains.
Simple model-based reinforcement learning outperforms model-free methods in complex tasks.
problem Lagging performance of model-based reinforcement learning agents in non-trivial environments.
method Combining soft value estimates with stochastic value gradients.
result Simple model-based agents achieve state-of-the-art results in a high-dimensional humanoid control task.
Adapts model-based advice to stabilize black-box policies for nonlinear control.
problem Stabilizing machine-learned policies for nonlinear control with limited model information.
method Proposes an adaptive -confident policy to combine black-box and model-based advice.
result Proves the stability of the adaptive -confident policy and its competitive ratio.
A framework schedules hyperparameters for model-based reinforcement learning, improving performance.
problem Inadequate scheduling of hyperparameters in model-based reinforcement learning.
method Theoretical analysis and AutoMBPO framework to automatically schedule real data ratio and other hyperparameters.
result Training with hyperparameters scheduled by AutoMBPO significantly improves performance.
The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy based on data samples. Although using many samples tends to improve the accuracy of policy learning, c…
Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization…
MOPO optimizes offline RL by penalizing dynamics uncertainty.
problem Learning policies from offline data with distributional shift.
method Modify model-based RL to avoid distributional shift.
result MOPO outperforms model-free and standard model-based RL.
Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In addition, they are typically based on the…
Paper proposes BMPO to optimize policies using bidirectional models.
problem Model-based reinforcement learning's reliance on forward model accuracy.
method Develops BMPO using both forward and backward models for policy optimization.
result BMPO outperforms state-of-the-art methods in sample efficiency and asymptotic performance.
This paper proposes a novel deep reinforcement learning architecture that was inspired by previous tree structured architectures which were only useable in discrete action spaces. Policy Prediction Network offers a way to improve sample complexity and performance on continuous control problems in exchange for extra com…
Unified reinforcement learning methods using hybrid inference.
problem Combining model-based and model-free reinforcement learning approaches.
method Control as Hybrid Inference (CHI) framework.
result CHI algorithm balances model-based and model-free learning.
A new reinforcement learning method uses model derivatives to improve policy optimization.
problem Improving sample efficiency and performance in model-based reinforcement learning.
method Constructs an actor-critic algorithm that uses the pathwise derivative of the learned model and policy.
result Consistently more sample efficient and matches model-free algorithms' asymptotic performance.
M3PO improves model-based meta-RL with theoretical guarantees.
problem Improving sample efficiency in multi-task RL with theoretical guarantees.
method Extending Janner et al. (2019) theorems, proposing M3PO with performance guarantees.
result M3PO outperforms existing methods in continuous-control benchmarks.
This paper tackles distribution shift in model-based offline RL, proposing a shifts-aware reward method.
problem Distribution shift challenges model-based offline RL by distorting value estimation and policy optimization.
method The paper disentangles the problem into model bias and policy shift, proposing a shifts-aware reward through probabilistic inference.
result The proposed shifts-aware reward method effectively mitigates distribution shift and improves policy optimization.
New model-based methods adapt pre-trained policies to unseen environments efficiently.
problem High sample complexity in reinforcement learning limits practical applications.
method Combines online learning and adaptive control to adapt policies in unseen environments.
result Proves policies can quickly recover trajectories from source to target environments.
Improves data efficiency in multi-agent control tasks using model-based reinforcement learning.
problem Limited data efficiency in reinforcement learning for multi-agent tasks.
method Decentralized model-based policy optimization (DMPO) framework.
result DMPO achieves superior data efficiency and matches model-free methods using true models.
Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-b…
New algorithms for model selection in off-policy evaluation of reinforcement learning.
problem Hyperparameter tuning for off-policy evaluation methods in reinforcement learning.
method Developed new model-free and model-based selectors with theoretical guarantees and a new experimental protocol.
result New model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
Model-Based Offline Planning (MBOP) learns models from offline data to control systems directly.
problem Training RL policies from offline data without direct system interaction.
method Generates models from offline data and uses planning to control the system.
result Near-optimal policies found for simulated systems with minimal real-time interaction.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Model-free reinforcement learning methods such as the Proximal Policy Optimization algorithm (PPO) have successfully applied in complex decision-making problems such as Atari games. However, these methods suffer from high variances and high sample complexity. On the other hand, model-based reinforcement learning method…
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
The most data-efficient algorithms for reinforcement learning in robotics are model-based policy search algorithms, which alternate between learning a dynamical model of the robot and optimizing a policy to maximize the expected return given the model and its uncertainties. However, the current algorithms lack an effec…
Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may …
Paper proposes a new framework to improve policy optimization by aligning real and simulated data distributions.
problem Inaccurate model estimation leads to performance degradation in model-based reinforcement learning.
method Introduces unsupervised model adaptation to minimize the IPM between real and simulated data distributions.
result Achieves state-of-the-art performance in sample efficiency on various continuous control tasks.
Proposes H-UCRL for efficient model-based RL with sublinear regret.
problem Greedy policy exploration in model-based RL ignores epistemic uncertainty.
method Reparameterizes plausible models, hallucinates control, augments input space, solves with greedy planners.
result H-UCRL achieves provably sublinear regret for Gaussian Process models.
BREMEN optimizes policies offline with fewer data, achieving efficient deployment.
problem High cost of updating policies in real-world applications.
method Behavior-Regularized Model-ENsemble (BREMEN) algorithm for offline optimization.
result BREMEN achieves impressive deployment efficiency with 5-10 deployments, outperforming standard RL methods.
The most data-efficient algorithms for reinforcement learning in robotics are model-based policy search algorithms, which alternate between learning a dynamical model of the robot and optimizing a policy to maximize the expected return given the model and its uncertainties. Among the few proposed approaches, the recent…
New RL method handles diverse human operator data.
problem Handling data from multiple, unknown policies.
method Domain-Invariant Model-based Offline RL (DIMORL) with Risk Extrapolation (REx).
result Models trained with REx generalize better across different demonstrators.
POLAR optimizes treatment strategies in dynamic settings with statistical guarantees.
problem Optimizing sequential decisions in dynamic treatment regimes with robustness and statistical guarantees.
method Pessimistic model-based approach estimating transition dynamics and incorporating uncertainty penalties.
result Offers statistical and computational guarantees, including finite-sample bounds on policy suboptimality.
Model-based reinforcement learning (RL) has proven to be a data efficient approach for learning control tasks but is difficult to utilize in domains with complex observations such as images. In this paper, we present a method for learning representations that are suitable for iterative model-based policy improvement, e…
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
Hybrid controller combines model-based and policy-based reinforcement learning.
problem Combining model-based and policy-based reinforcement learning for stability and robustness.
method Designs a hybrid controller that interpolates a model-based linear controller and a differentiable policy.
result Proven to maintain stability and universal approximation properties.
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
New bounds assess policy evaluation under unobserved confounders, showing model-based methods are more effective.
problem Policy evaluation under unobserved confounders in uncertain causal environments.
method Developed worst-case bounds for sensitivity to unobserved confounders, demonstrating model-based methods are more effective.
result Model-based approaches with robust MDPs provide sharper lower bounds for policy evaluation.
FOCUS improves offline RL by incorporating causal structure into world-models.
problem Learning effective policies from historical data without interaction.
method FOCUS proposes a practical algorithm that learns and leverages causal structure in offline RL.
result FOCUS outperforms plain model-based offline RL algorithms and other causal model-based RL algorithms.
CPPO learns policies from partial offline data in MDPs with structural assumptions.
problem Offline Reinforcement Learning with partial coverage assumption.
method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.
result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.
We introduce Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects. In contrast to most existing model-based reinforcement learning and planning methods, which prescribe how a model should be used to arrive at a policy, I2As learn to inter…
A method to reduce bias in model-based policy evaluation by shifting operators.
problem Bias in value function computation from noisy estimated models.
method Operator shifting method to reduce the residual norm error.
result The shifting factor is always positive and upper bounded by $1+O\left(1/n
ight)$.
Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate s…
The paper proposes an iterative approach to batch reinforcement learning for safer and more informative data collection.
problem Learning policies that are too rigid and do not adapt to new data.
method Safe diversified model-based policy search in an iterative batch reinforcement learning framework.
result Improved learned policies through continuous data collection and adaptation.
PriMORL trains private RL policies on offline data.
problem Private reinforcement learning on offline data.
method PriMORL learns DP models of the environment and optimizes a policy on the penalized private model.
result PriMORL enables training of private RL agents on complex tasks.