Hybrid controller combines model-based and policy-based reinforcement learning.
problem Combining model-based and policy-based reinforcement learning for stability and robustness.
method Designs a hybrid controller that interpolates a model-based linear controller and a differentiable policy.
result Proven to maintain stability and universal approximation properties.
Simple model-based reinforcement learning outperforms model-free methods in complex tasks.
problem Lagging performance of model-based reinforcement learning agents in non-trivial environments.
method Combining soft value estimates with stochastic value gradients.
result Simple model-based agents achieve state-of-the-art results in a high-dimensional humanoid control task.
Recent success in deep reinforcement learning for continuous control has been dominated by model-free approaches which, unlike model-based approaches, do not suffer from representational limitations in making assumptions about the world dynamics and model errors inevitable in complex domains. However, they require a lo…
Adapts model-based advice to stabilize black-box policies for nonlinear control.
problem Stabilizing machine-learned policies for nonlinear control with limited model information.
method Proposes an adaptive λ λ λ -confident policy to combine black-box and model-based advice. result Proves the stability of the adaptive λ λ λ -confident policy and its competitive ratio. Unified reinforcement learning methods using hybrid inference.
problem Combining model-based and model-free reinforcement learning approaches.
method Control as Hybrid Inference (CHI) framework.
result CHI algorithm balances model-based and model-free learning.
This paper compares model-based and model-free control methods using neural networks.
problem Comparing model-based and model-free control methods for unknown nonlinear systems.
method Utilizes Deep Koopman Representation (DKRC) and Deep Deterministic Policy Gradient (DDPG) for control.
result DKRC outperforms DDPG in terms of control strategies and accuracy for unknown dynamics.
Model based predictions of future trajectories of a dynamical system often suffer from inaccuracies, forcing model based control algorithms to re-plan often, thus being computationally expensive, suboptimal and not reliable. In this work, we propose a model agnostic method for estimating the uncertainty of a model?s pr…
Policy Prediction Network improves continuous control problems with model-free and model-based learning.
problem Improving sample complexity and performance in continuous control problems.
method Integrates model-free and model-based reinforcement learning, introduces implicit model-based learning for continuous action space.
result First to introduce implicit model-based learning to Policy Gradient algorithms for continuous action space.
New model learning objective improves continuous control tasks.
problem Challenges in solving continuous control tasks using model-based reinforcement learning.
method Derived a novel value-aware model learning objective and identified and addressed stale value estimates issue.
result Value-aware objectives can be successfully deployed in solving continuous control tasks without tuning hyper-parameters.
Continuous-time MBRL framework tackles control systems with Bayesian ODEs.
problem Discretization of continuous-time systems in MBRL.
method Novel actor-critic method with Bayesian ODEs for state inference.
result Model robust against irregular and noisy data, sample-efficient, solves challenging control problems.
Deep reinforcement learning controls drones without model knowledge.
problem Real-time robot control without engineered models.
method Learnt probabilistic model of drone dynamics, model-based reinforcement learning.
result Controller and value function optimized through generated latent trajectories.
The effectiveness of model-based versus model-free methods is a long-standing question in reinforcement learning (RL). Motivated by recent empirical success of RL on continuous control tasks, we study the sample complexity of popular model-based and model-free algorithms on the Linear Quadratic Regulator (LQR). We show…
Unified framework for policy improvement in RL with benefits in data efficiency and computation.
problem Improving data efficiency and computation in reinforcement learning for continuous control.
method Local, regularized policy improvement with tree search for continuous action spaces.
result Improves data efficiency and reduces wall-clock time in high-dimensional domains.
Improves data efficiency in multi-agent control tasks using model-based reinforcement learning.
problem Limited data efficiency in reinforcement learning for multi-agent tasks.
method Decentralized model-based policy optimization (DMPO) framework.
result DMPO achieves superior data efficiency and matches model-free methods using true models.
New algorithm minimizes worst-case regret in uncertain, time-varying dynamics.
problem Model-based policy learning in uncertain, time-varying dynamics.
method Planning regret metric and iterative algorithm for minimizing it.
result Empirical evidence shows the proposed algorithm outperforms existing methods.
Proposes a new reinforcement learning method to improve agent performance in control tasks.
problem Shortcomings of maximum likelihood estimation in model-based reinforcement learning.
method Directly optimizes expected returns using implicit differentiation of a Bellman optimality function.
result Empirical evidence shows improved performance in model misspecification regime.
A new search-control strategy improves Dyna's efficiency.
problem Improving sample efficiency in model-based reinforcement learning.
method Proposes a novel search-control strategy by sampling high frequency regions of the value function.
result Empirically shows that high frequency regions require more samples to approximate, suggesting a better search-control strategy.
PID control architectures are widely used in industrial applications. Despite their low number of open parameters, tuning multiple, coupled PID controllers can become tedious in practice. In this paper, we extend PILCO, a model-based policy search framework, to automatically tune multivariate PID controllers purely bas…
Combines MFRL and MPC for data-efficient high performance in robotics.
problem Lack of data efficiency in MBRL for real-world applications.
method Combines Model-Free Reinforcement Learning (MFRL) and Model Predictive Control (MPC).
result Achieves MFRL's performance level with MBRL's data efficiency.
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
Optimizes reinforcement learning controllers for reliability.
problem Lack of robustness in reinforcement learning controllers due to deterministic reward.
method Introduces a reliability-based optimization framework using a model-based approach.
result Stable controllers learned in static and dynamic environments on classical tasks.
Model-Based Offline Planning (MBOP) learns models from offline data to control systems directly.
problem Training RL policies from offline data without direct system interaction.
method Generates models from offline data and uses planning to control the system.
result Near-optimal policies found for simulated systems with minimal real-time interaction.
M3PO improves model-based meta-RL with theoretical guarantees.
problem Improving sample efficiency in multi-task RL with theoretical guarantees.
method Extending Janner et al. (2019) theorems, proposing M3PO with performance guarantees.
result M3PO outperforms existing methods in continuous-control benchmarks.
POPLIN improves model-based planning in complex environments.
problem Efficient planning in complex high-dimensional environments.
method Combines policy networks with online planning, optimizing parameters directly.
result POPLIN achieves state-of-the-art performance in MuJoCo benchmarks, 3x more sample efficient.
Proposes a new framework to optimize policies while accounting for model uncertainty.
problem Model-based reinforcement learning's vulnerability to model bias and inefficiency.
method Uncertainty-aware model-based policy optimization framework.
result Significantly lower sample complexity and competitive asymptotic performance.
A framework for data-driven decision-making in infectious disease control.
problem Optimizing trade-offs between public health and economic impacts.
method Multi-objective model-based reinforcement learning.
result Pareto-optimal policies minimizing long-term costs.
Paper proposes CARL for better control in RL from sensory data.
problem Efficient control of complex systems from high-dimensional sensory inputs.
method Control-Aware Representation Learning (CARL) for model-based RL.
result CARL improves control performance in benchmark tasks.
Unified Latent Dynamics unifies model-free and model-based reinforcement learning.
problem Combining the efficiency of model-free methods with the representational strengths of model-based approaches.
method Embedding state-action pairs into a latent space where the true value function is approximately linear, using synchronized updates of encoder, value, and policy networks.
result ULD achieves cross-domain competence with minimal tuning and a fraction of the parameter footprint.
Regularizes model-based planning using energy-based models for efficient learning.
problem Challenges in using learned dynamics models for accurate planning.
method Regularization using energy estimates of state transitions.
result Proposes effective regularization method for planning with pre-trained dynamics models.
New model-based methods adapt pre-trained policies to unseen environments efficiently.
problem High sample complexity in reinforcement learning limits practical applications.
method Combines online learning and adaptive control to adapt policies in unseen environments.
result Proves policies can quickly recover trajectories from source to target environments.
Unified framework for model-based RL with sample complexity guarantees.
problem Designing efficient posterior sampling methods for model-based RL.
method Optimistic posterior sampling, Hellinger distance reduction, data likelihood measurement.
result Unified algorithms with state-of-the-art sample complexity guarantees.
Proposes H-UCRL for efficient model-based RL with sublinear regret.
problem Greedy policy exploration in model-based RL ignores epistemic uncertainty.
method Reparameterizes plausible models, hallucinates control, augments input space, solves with greedy planners.
result H-UCRL achieves provably sublinear regret for Gaussian Process models.
Digital twins improve single-arm trials by providing robust treatment effect estimates.
problem Lack of control arms in single-arm trials limits their gold-standard evidence.
method Outcome-model-based synthetic controls using machine learning models trained on historical data.
result Digital twins offer more robust treatment effect estimates and principled corrections.
VMBPO optimizes model and policy jointly using variational lower-bound.
problem Data efficiency in RL with biased simulated data.
method Formulate variational objective function, use EM, iteratively improve model and policy.
result VMBPO is more sample-efficient and robust than model-free algorithms.
Proposes a method to control model complexity in neural network optimization.
problem Reduces the computational cost of neural architecture search.
method Probabilistic model-based dynamic optimization with a penalty term to control model complexity.
result The proposed method controls model complexity while maintaining performance.
FLEX optimizes exploration for nonlinear systems with minimal data.
problem Efficient exploration of unknown nonlinear systems with limited data.
method FLEX uses optimal experimental design to maximize information gain.
result FLEX outperforms other methods in nonlinear environments and control tasks.
This paper improves robot grasping by integrating meta-control and latent-space imagination.
problem Dual-system approaches fail to consider the reliability of the learned model when making multiple-step predictions.
method A meta-controller arbitrates between model-based and model-free decisions based on local reliability, encouraging actions that improve the model and generating imagined experiences for additional training.
result Our approach learns near-optimal grasping policies in dense- and sparse-reward environments, outperforming baseline and state-of-the-art methods.
Paper identifies objective mismatch in MBRL, affecting control task performance.
problem Objective mismatch in MBRL framework affects control task performance.
method Proposes re-weighting dynamics model training to mitigate mismatch.
result Likelihood of one-step ahead predictions is not always correlated with control performance.
Minimum attention improves reinforcement learning performance in high-dimensional dynamics.
problem Improving reinforcement learning performance in high-dimensional nonlinear dynamics.
method Applying minimum attention as a regularization technique in reinforcement learning, including model-based and model-free approaches.
result Minimum attention outperforms state-of-the-art algorithms in few-shot adaptation and variance reduction.
AOP combines model-based planning with model-free learning to handle lifelong learning challenges.
problem Learning control in an online reset-free lifelong learning scenario where mistakes can compound and dynamics change.
method Adaptive Online Planning (AOP) that combines model-based planning with model-free learning, approximating uncertainty to call upon planning only when necessary.
result Achieves strong performance in lifelong learning challenges, gracefully adapting behaviors in the face of unpredictable changes.
A framework schedules hyperparameters for model-based reinforcement learning, improving performance.
problem Inadequate scheduling of hyperparameters in model-based reinforcement learning.
method Theoretical analysis and AutoMBPO framework to automatically schedule real data ratio and other hyperparameters.
result Training with hyperparameters scheduled by AutoMBPO significantly improves performance.
Proposes a method to learn dynamic models for systems with variable number of objects.
problem Efficiently modeling systems with a variable number of objects.
method Uses graph neural networks and block-wise linear transition matrices to learn compositional Koopman operators.
result The method adapts to new environments and produces better control signals.
Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of c…
Optimistic RL algorithms are simplified for deep RL with competitive performance.
problem Achieving accurate optimism in model-based RL for large-scale problems.
method Interpreting scalable optimistic model-based algorithms as solving a tractable noise augmented MDP.
result Competitive regret bound of i l d e O ( ∣ S ∣ H ∣ A ∣ T ) ilde{\mathcal{O}}( |\mathcal{S}|H\sqrt{|\mathcal{A}| T } ) i l d e O ( ∣ S ∣ H ∣ A ∣ T ) for Gaussian noise augmentation. Combining causality, control, and reinforcement learning for system control.
problem Learning to control dynamical systems using causal, control, and reinforcement learning approaches.
method Combining causal identification, control strategies, and reinforcement learning to control dynamical systems.
result Combining different learning paradigms for effective system control.
PriMORL trains private RL policies on offline data.
problem Private reinforcement learning on offline data.
method PriMORL learns DP models of the environment and optimizes a policy on the penalized private model.
result PriMORL enables training of private RL agents on complex tasks.
CTRL improves reinforcement learning by combining control strategies.
problem Controlling agents to herd and contain target agents in a goal region.
method CTRL algorithm that enhances tabular learning algorithms with a tutoring model-based control strategy.
result Significantly reduces learning times and improves exploration of the state-space.
Reinforcement learning algorithms are gaining popularity in fields in which optimal scheduling is important, and oncology is not an exception. The complex and uncertain dynamics of cancer limit the performance of traditional model-based scheduling strategies like Optimal Control. Motivated by the recent success of mode…