Survey of integrating planning and learning in model-based reinforcement learning.
problem Sequential decision making in AI, formalized as MDP optimization.
method Systematic coverage of dynamics model learning and planning-learning integration.
result Broad conceptual overview of model-based reinforcement learning.
Greedy policies in model-based RL achieve tight regret bounds without full planning.
problem Achieving efficient RL algorithms in MDP settings.
method Using greedy policies for 1-step planning in model-based RL.
result Greedy policies achieve i l d e O ( H S A T ) ilde{\mathcal{O}}(\sqrt{HSAT}) i l d e O ( H S A T ) regret bounds. DPN combines model-based and model-free reinforcement learning for efficient planning.
problem Efficiently plan actions in reinforcement learning environments.
method Combines model-based and model-free reinforcement learning, dynamically constructing plans using a learned state-transition model.
result Reduces the number of state transitions during planning by up to 96%, improving data efficiency and performance.
Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the "Imagination-based Planner", the first m…
The paper tackles model-based RL's inaccuracy issue by dynamically adjusting planning horizons.
problem Model-based RL's failure due to model inaccuracy over long planning horizons.
method State-dependent planning horizon, learning cumulative model errors with Temporal Difference methods.
result The proposed method successfully adapts planning horizons to state-dependent model accuracy, improving policy learning efficiency.
EBMs improve sample efficiency and generalization in RL.
problem Improving sample efficiency and generalization in reinforcement learning.
method Developed an online algorithm to train EBMs for model-based planning, leveraging their ability to infer intermediate states.
result EBMs lead to significantly better online learning and state space planning compared to feed-forward networks.
Proposes learning latent reward model for planning from rewards.
problem Planning in high-dimensional state spaces with limited reward information.
method Directly learns a latent dynamics model from rewards, planning in latent state-space.
result Successfully learns accurate latent reward prediction model, achieving strong performance and high sample efficiency.
AOP combines model-based planning with model-free learning to handle lifelong learning challenges.
problem Learning control in an online reset-free lifelong learning scenario where mistakes can compound and dynamics change.
method Adaptive Online Planning (AOP) that combines model-based planning with model-free learning, approximating uncertainty to call upon planning only when necessary.
result Achieves strong performance in lifelong learning challenges, gracefully adapting behaviors in the face of unpredictable changes.
Regularizes model-based planning using energy-based models for efficient learning.
problem Challenges in using learned dynamics models for accurate planning.
method Regularization using energy estimates of state transitions.
result Proposes effective regularization method for planning with pre-trained dynamics models.
POPLIN improves model-based planning in complex environments.
problem Efficient planning in complex high-dimensional environments.
method Combines policy networks with online planning, optimizing parameters directly.
result POPLIN achieves state-of-the-art performance in MuJoCo benchmarks, 3x more sample efficient.
End-to-end Dyna-AIL combines planning and learning for faster convergence.
problem Efficient imitation learning with minimal interactions.
method Dyna-like framework integrating model-based planning and model-free learning.
result Our approach converges to optimal policies with fewer interactions.
New method uses autoregressive models for dynamic planning.
problem Planning in dynamic environments with moving obstacles and goals.
method Conditional autoregressive generative models in a discrete latent space.
result Method nearly matches true environment performance for planning.
Improves model-based control and exploration by estimating model uncertainty.
problem Inaccuracies in model predictions lead to frequent re-planning, inefficiency, and unreliability.
method Estimates model uncertainty using reconstruction error and uses it for better control and active exploration.
result Improves control performance and exploration efficiency by choosing confident model predictions and planning for high uncertainty.
Proposes a method for model-based RL in complex environments without perfect simulators.
problem Lack of cheap and perfect simulators in real-world tasks.
method Induces a world program by learning dynamics and actions in graph-based environments.
result World program enables complex planning tasks in environments without perfect simulators.
This paper calibrates deep reinforcement learning models to improve planning and performance.
problem Inaccurate predictive uncertainties from deep learning systems hinder model-based reinforcement learning.
method Describes a simple method to calibrate uncertainties in model-based reinforcement learning agents.
result Calibrated model-based reinforcement learning agents achieve state-of-the-art performance with fewer samples.
PALM learns abstract models for efficient planning and task transfer.
problem Efficiently learning and transferring hierarchical models for planning.
method PALM uses a new formal structure (L-AMDP) to learn independent, modular models at multiple levels of abstraction.
result PALM integrates planning and execution, facilitating rapid learning of abstract models.
Model-Based Offline Planning (MBOP) learns models from offline data to control systems directly.
problem Training RL policies from offline data without direct system interaction.
method Generates models from offline data and uses planning to control the system.
result Near-optimal policies found for simulated systems with minimal real-time interaction.
Paper proposes using expectation models for planning in stochastic environments.
problem Intractability of learning distribution and sample models in large state and action spaces.
method Proposes using approximate expectation models for MBRL, analyzes linear and non-linear parametrizations, and presents a policy evaluation algorithm.
result Planning with an expectation model is equivalent to planning with a distribution model under certain conditions.
Selective planning with imperfect models reduces harmful effects of model inadequacy.
problem Harmful effects of using an imperfect model in reinforcement learning.
method Selective planning with heteroscedastic regression to estimate predictive uncertainty from model inadequacy.
result Effective selective planning requires considering both parameter uncertainty and model inadequacy.
Study shows plug-in model-based reinforcement learning is minimax optimal.
problem Finding optimal policies in MDPs with limited generative model access.
method Developed and analyzed a plug-in approach to model-based reinforcement learning.
result Plug-in approach yields minimax optimal policies with sublinear sample complexity.
Residual algorithms improve reinforcement learning performance.
problem Distribution mismatch in model-based planning.
method Bidirectional target network technique for residual algorithms.
result Residual reinforcement learning significantly outperforms vanilla methods.
DADS discovers skills with predictable outcomes from unlabeled data.
problem Learning accurate models for complex dynamical systems is difficult and often doesn't generalize well.
method Dynamics-Aware Discovery of Skills (DADS) combines model-based and model-free learning.
result DADS discovers infinitely many behaviors in high-dimensional state-spaces.
PlaNet learns latent dynamics from images for better planning in unknown environments.
problem Leveraging planning in unknown environments with accurate dynamics models.
method Deep Planning Network (PlaNet) learns dynamics from images using latent space and multi-step variational inference.
result PlaNet achieves high performance in continuous control tasks with contact dynamics and sparse rewards.
Unified reinforcement learning methods using hybrid inference.
problem Combining model-based and model-free reinforcement learning approaches.
method Control as Hybrid Inference (CHI) framework.
result CHI algorithm balances model-based and model-free learning.
POLO framework enables efficient learning and exploration in model-based control.
problem Efficient learning and exploration in model-based control settings.
method Combines local model-based control, global value function learning, and exploration.
result POLO framework accelerates value function learning and enables better policies.
We introduce Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects. In contrast to most existing model-based reinforcement learning and planning methods, which prescribe how a model should be used to arrive at a policy, I2As learn to inter…
This paper builds a model to predict the long-term future in reinforcement learning.
problem Catastrophic failures due to flawed long-term predictions in reinforcement learning models.
method The authors develop a latent-variable autoregressive model using variational inference to incorporate future information.
result The model achieves higher rewards faster than baselines on various tasks and environments.
Study non-stationary MDPs using worst-case RL, proposing RATS algorithm.
problem Robust zero-shot planning in non-stationary stochastic environments.
method Model-Based Reinforcement Learning, worst-case approach.
result RATS algorithm demonstrates benefits over reference methods.
Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI. This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities. We propose a hybrid …
The study compares reinforcement learning models and finds model-based approaches superior for complex MDPs.
problem Complexity of optimal Q-functions and policies in MDPs exceeds dynamics, hindering model-free methods.
method Theoretical analysis and empirical testing of neural network expressivity for policies, Q-functions, and dynamics.
result Model-based planning yields better policies for complex MDPs, improving performance on MuJoCo tasks.
New approach handles stochastic and partially-observable environments using discrete autoencoders and Monte Carlo tree search.
problem Challenges in planning for stochastic and partially-observable environments.
method Uses discrete autoencoders and a stochastic variant of Monte Carlo tree search.
result Significantly outperforms MuZero on stochastic chess and scales to DeepMind Lab.
Thompson Sampling improves pursuit-evasion coordination.
problem Optimizing pursuer behavior to capture an evader efficiently.
method Variant of Thompson Sampling applied to pursuit-evasion problems.
result Thompson Sampling reduces time-to-capture significantly.
This work learns visual representations for deformable objects using contrastive estimation.
problem Challenges in learning plannable visual representations for deformable objects.
method Jointly optimizes visual representation and dynamics models using contrastive estimation.
result Substantial improvements in performance over standard model-based learning techniques.
PlanGAN uses GANs to plan efficient trajectories for multi-goal tasks in sparse reward environments.
problem Learning with sparse rewards in multi-goal environments.
method PlanGAN combines GANs to generate trajectories leading to specified goals, then combines these into a planning algorithm.
result PlanGAN achieves comparable performance to model-free RL but is 4-8 times more sample efficient.
Improved CEM for fast real-time planning in high-dimensional control tasks.
problem Sampling inefficiency of CEM in real-time planning.
method Novel additions to CEM including temporally-correlated actions and memory.
result 2.7-22x less samples and 1.2-10x performance increase.
The computational costs of inference and planning have confined Bayesian model-based reinforcement learning to one of two dismal fates: powerful Bayes-adaptive planning but only for simplistic models, or powerful, Bayesian non-parametric models but using simple, myopic planning strategies such as Thompson sampling. We …
A multi-step model reduces compounding errors in reinforcement learning.
problem Compounding errors in one-step models lead to inaccurate predictions in reinforcement learning.
method Introduced a multi-step model that directly outputs the outcome of a sequence of actions.
result The multi-step model yields better action selection and more accurate value-function estimation.
New algorithm minimizes worst-case regret in uncertain, time-varying dynamics.
problem Model-based policy learning in uncertain, time-varying dynamics.
method Planning regret metric and iterative algorithm for minimizing it.
result Empirical evidence shows the proposed algorithm outperforms existing methods.
This paper presents a unifying framework for reinforcement learning and planning.
problem Sequential decision making in AI, formalized as MDP optimization.
method A unifying algorithmic framework (FRAP) for reinforcement learning and planning.
result Identifies common dimensions in MDP planning and learning algorithms.
Unified Latent Dynamics unifies model-free and model-based reinforcement learning.
problem Combining the efficiency of model-free methods with the representational strengths of model-based approaches.
method Embedding state-action pairs into a latent space where the true value function is approximately linear, using synchronized updates of encoder, value, and policy networks.
result ULD achieves cross-domain competence with minimal tuning and a fraction of the parameter footprint.
Novel approach to learning models based on subjective timescales for better exploration and decision-making.
problem Learning models over multi-step timescales in environments with intermediate states.
method Developed a subjective-timescale model (STM) based on episodic memories, enabling systematic variation of temporal extent of predictions.
result STM produces more informative action-conditioned roll-outs, leading to better decision-making and exploration.
A new multi-step model improves model-based reinforcement learning efficiency.
problem Expensive environmental interaction in reinforcement learning.
method Proposes a multi-step model for predicting action sequences with variable length.
result Multi-step model outperforms one-step model in preliminary tests.
Paper benchmarks MBRL algorithms with new environments.
problem Lack of standardized benchmarking for MBRL algorithms.
method Proposes 18 new benchmarking environments and standardizes problem settings.
result Characterizes three key research challenges for MBRL.
We designed a grid world task to study human planning and re-planning behavior in an unknown stochastic environment. In our grid world, participants were asked to travel from a random starting point to a random goal position while maximizing their reward. Because they were not familiar with the environment, they needed…
This paper presents preliminary work on learning the search heuristic for the optimal motion planning for automated driving in urban traffic. Previous work considered search-based optimal motion planning framework (SBOMP) that utilized numerical or model-based heuristics that did not consider dynamic obstacles. Optimal…
Model learns and plans in real-time under constraints for robotic systems.
problem Challenges in applying reinforcement learning to robotic systems, especially handling continuous state and action spaces, time/resource budget, and hard constraints.
method Combines Gaussian Process regression and Receding Horizon Control. Uses sparse spectrum Gaussian Processes for incremental model updates from sensory data.
result Demonstrates benefits of online learning on autonomous racing tasks and reusability of learned dynamics.
Enhances PlaNet for better planning in uncertain environments.
problem Improving deep planning networks for partially observable environments.
method Incorporates Bayesian inference to handle uncertainty in latent models and action candidates.
result Consistently improves asymptotic performance on continuous control tasks.
Self-consistent models improve reinforcement learning by aligning predictions with future values.
problem Improving reinforcement learning by aligning model predictions with future values.
method Proposes multiple self-consistency updates to encourage a learned model and value function to be consistent with each other.
result Self-consistency helps both policy evaluation and control in both tabular and function approximation settings.