Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, mode…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “Model-Based Exploration”
Proposes H-UCRL for efficient model-based RL with sublinear regret.
problem Greedy policy exploration in model-based RL ignores epistemic uncertainty.
method Reparameterizes plausible models, hallucinates control, augments input space, solves with greedy planners.
result H-UCRL achieves provably sublinear regret for Gaussian Process models.
Model-free reinforcement learning methods such as the Proximal Policy Optimization algorithm (PPO) have successfully applied in complex decision-making problems such as Atari games. However, these methods suffer from high variances and high sample complexity. On the other hand, model-based reinforcement learning method…
FLEX optimizes exploration for nonlinear systems with minimal data.
problem Efficient exploration of unknown nonlinear systems with limited data.
method FLEX uses optimal experimental design to maximize information gain.
result FLEX outperforms other methods in nonlinear environments and control tasks.
DQN with model-based exploration improves learning in sparse reward environments.
problem Poor sample efficiency in sparse reward environments.
method Combines model-free and model-based approaches for better exploration.
result Improves performance in environments with sparse rewards.
Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimizationcs.LG
Model based predictions of future trajectories of a dynamical system often suffer from inaccuracies, forcing model based control algorithms to re-plan often, thus being computationally expensive, suboptimal and not reliable. In this work, we propose a model agnostic method for estimating the uncertainty of a model?s pr…
This paper calibrates deep reinforcement learning models to improve planning and performance.
problem Inaccurate predictive uncertainties from deep learning systems hinder model-based reinforcement learning.
method Describes a simple method to calibrate uncertainties in model-based reinforcement learning agents.
result Calibrated model-based reinforcement learning agents achieve state-of-the-art performance with fewer samples.
Simple method improves exploration in various decision problems.
problem Improving exploration in sequential decision problems.
method Parameterized Exploration (PE) method that considers time horizon and state of knowledge.
result PE outperforms un-tuned methods in various bandit and decision problem settings.
A new algorithm STE for model-based RL improves learning rates.
problem Sparse rewards and computational intractability of estimating information gain.
method Developed a novel algorithm based on Stein Information Directed Exploration (STE)E.
result Achieves sublinear Bayesian regret, outperforming prior approaches.
Efficient exploration is an unsolved problem in Reinforcement Learning which is usually addressed by reactively rewarding the agent for fortuitously encountering novel situations. This paper introduces an efficient active exploration algorithm, Model-Based Active eXploration (MAX), which uses an ensemble of forward mod…
This paper tackles hard exploration in the game Pommerman, improving RL learning.
problem Hard exploration in sparse, delayed, and deceptive reward domains.
method Developed a model-based automatic reasoning module to prune unsafe actions.
result Model-based approach significantly improves RL learning in Pommerman.
The most data-efficient algorithms for reinforcement learning in robotics are model-based policy search algorithms, which alternate between learning a dynamical model of the robot and optimizing a policy to maximize the expected return given the model and its uncertainties. However, the current algorithms lack an effec…
EBMs improve sample efficiency and generalization in RL.
problem Improving sample efficiency and generalization in reinforcement learning.
method Developed an online algorithm to train EBMs for model-based planning, leveraging their ability to infer intermediate states.
result EBMs lead to significantly better online learning and state space planning compared to feed-forward networks.
The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy based on data samples. Although using many samples tends to improve the accuracy of policy learning, c…
New method quantifies uncertainty in reinforcement learning models.
problem Quantifying uncertainty over expected cumulative rewards in reinforcement learning.
method Proposes a new uncertainty Bellman equation to more accurately estimate value function variance.
result Our method converges to the true posterior variance over values and improves sample-efficiency.
This paper analyzes how randomizing rewards in MBRL can improve performance without being overly optimistic.
problem The gap between theoretical worst-case regret analysis and empirical performance in MBRL.
method Reward randomization in model-based reinforcement learning (MBRL) with kernelized linear regulator (KNR) model.
result Reward randomization guarantees partial optimism and near-optimal worst-case regret.
GIM learns by analogy to improve reinforcement learning efficiency.
problem Efficient reinforcement learning with limited data.
method GIM infers unknown dynamics from known dynamics based on spectral properties.
result GIM achieves lower sample complexity and computational efficiency.
RL in MFGs is as hard as solving many single-agent RL problems.
problem Learning Nash Equilibrium in Mean-Field Games (MFGs).
method Introduce P-MBED to measure model complexity, develop a novel exploration strategy, and establish polynomial sample complexity results.
result Learning Nash Equilibrium in MFGs is no more statistically challenging than solving a logarithmic number of single-agent RL problems.
Paper proposes a new approach for agents to explore environments efficiently.
problem Learning policies that explore uniformly and mix quickly in environments without external rewards.
method Introduces a surrogate objective to maximize entropy and develops a model-based reinforcement learning algorithm, IDEAL.
result Demonstrates improved exploration and mixing in hard-exploration tasks.
Novel approach to learning models based on subjective timescales for better exploration and decision-making.
problem Learning models over multi-step timescales in environments with intermediate states.
method Developed a subjective-timescale model (STM) based on episodic memories, enabling systematic variation of temporal extent of predictions.
result STM produces more informative action-conditioned roll-outs, leading to better decision-making and exploration.
We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local traj…
Greedy policies in model-based RL achieve tight regret bounds without full planning.
problem Achieving efficient RL algorithms in MDP settings.
method Using greedy policies for 1-step planning in model-based RL.
result Greedy policies achieve regret bounds.
We study the sample complexity of model-based reinforcement learning (henceforth RL) in general contextual decision processes that require strategic exploration to find a near-optimal policy. We design new algorithms for RL with a generic model class and analyze their statistical properties. Our algorithms have sample …
Curious Replay improves model-based reinforcement learning agents' adaptability.
problem Existing model-based reinforcement learning agents struggle to adapt quickly to changing environments.
method Curious Replay uses a curiosity-based priority signal for prioritized experience replay tailored to model-based agents.
result Agents using Curious Replay achieve improved performance in exploration and on benchmarks.
SAVED safely learns robot tasks with sparse rewards.
problem Challenges in reinforcement learning for robotics, especially sparse rewards and complex constraints.
method SAVED uses supervision to constrain exploration and learn efficiently, handling complex constraints.
result SAVED outperforms state-of-the-art methods in success rate, constraint satisfaction, and sample efficiency.
Asynchronous framework speeds up model-based RL to real-time.
problem Real-time learning on real robots with model-based RL.
method Asynchronous framework for model-based reinforcement learning.
result Reduced run time to data collection time, improved sample complexity.
Bayesian model-based reinforcement learning is a formally elegant approach to learning optimal behaviour under model uncertainty, trading off exploration and exploitation in an ideal way. Unfortunately, finding the resulting Bayes-optimal policies is notoriously taxing, since the search space becomes enormous. In this …
Sepsis is a dangerous condition that is a leading cause of patient mortality. Treating sepsis is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. In this work, we explore the use of continuous state-space mode…
Active inference implemented for high-dimensional tasks shows efficient exploration and improved sample efficiency.
problem Achieving efficient exploration and learning in complex, uncertain environments.
method Active inference framework applied to high-dimensional tasks, with Bayesian evidence maximization.
result Order of magnitude increase in sample efficiency over model-free baselines.
Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.
problem Challenging to learn accurate MDPs for high-dimensional states.
method Learn an abstract MDP over low-dimensional coarse states, using an abstraction function.
result Achieves superhuman performance on Pitfall! and higher reward with fewer samples.
RP1 uses active learning to improve world model in fewest samples.
problem Improving sample efficiency in MBRL for continuous control tasks.
method RP1 views MBRL as active learning, using a hybrid objective function and principled termination.
result Statistically significant gains over existing approaches on continuous control tasks.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
Optimizes exploration for nonlinear systems to learn controllers efficiently.
problem Learning optimal controllers for unknown nonlinear systems.
method Formally quantifies which parameters are most critical, and develops an algorithm to efficiently explore these parameters.
result Proves a near-instance-optimal rate for learning controllers.
Combines MFRL and MPC for data-efficient high performance in robotics.
problem Lack of data efficiency in MBRL for real-world applications.
method Combines Model-Free Reinforcement Learning (MFRL) and Model Predictive Control (MPC).
result Achieves MFRL's performance level with MBRL's data efficiency.
Improved exploration algorithm for unknown MDPs with reduced sample complexity.
problem Exploration of unknown environments without reward function.
method Incremental model-based approach that interleaves state discovery and policy improvement.
result Achieves sample complexity scaling as .
Paper benchmarks MBRL algorithms with new environments.
problem Lack of standardized benchmarking for MBRL algorithms.
method Proposes 18 new benchmarking environments and standardizes problem settings.
result Characterizes three key research challenges for MBRL.
MOPO optimizes offline RL by penalizing dynamics uncertainty.
problem Learning policies from offline data with distributional shift.
method Modify model-based RL to avoid distributional shift.
result MOPO outperforms model-free and standard model-based RL.
Novelty search in low-dimensional space improves sample efficiency in exploration tasks.
problem Efficient exploration in complex environments with sparse rewards.
method Combines model-based and model-free objectives to learn a low-dimensional representation. Uses intrinsic novelty rewards based on nearest neighbor distances in this space.
result Our approach achieves more sample-efficient exploration compared to strong baselines on various tasks.
CARL safely adapts RL agents for safety-critical tasks.
problem Safety hazards in RL for safety-critical tasks.
method CARL combines model-based RL and cautious adaptation.
result CARL achieves higher rewards with fewer failures in safety-critical tasks.
New RL algorithm for large state spaces with explicit exploration and exploitation phases.
problem Reinforcement learning in large or infinite state spaces.
method Model-based approach with explicit exploration and exploitation phases.
result Proves near-optimal policy with polynomial sample complexity under certain assumptions.
New algorithm learns policies without explicit rewards for MDPs.
problem Learning policies for MDPs without explicit rewards.
method UCRL-RFE algorithm with linear function approximation.
result UCRL-RFE achieves optimal policy with minimal exploration samples.
A new deep learning model improves phase retrieval performance.
problem Recovering signals from phaseless measurements.
method Hybrid model-based data-driven deep architecture (Unfolded Phase Retrieval, UPR).
result Significant improvement in phase retrieval performance.
Minimum attention improves reinforcement learning performance in high-dimensional dynamics.
problem Improving reinforcement learning performance in high-dimensional nonlinear dynamics.
method Applying minimum attention as a regularization technique in reinforcement learning, including model-based and model-free approaches.
result Minimum attention outperforms state-of-the-art algorithms in few-shot adaptation and variance reduction.
In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuse…
Optimistic RL algorithms are simplified for deep RL with competitive performance.
problem Achieving accurate optimism in model-based RL for large-scale problems.
method Interpreting scalable optimistic model-based algorithms as solving a tractable noise augmented MDP.
result Competitive regret bound of for Gaussian noise augmentation.
OMLE combines optimism and MLE for efficient sequential decision making.
problem Efficiently solving sequential decision making problems, especially in partially observable settings.
method Combines optimism for exploration and maximum likelihood estimation for model learning.
result OMLE learns near-optimal policies for a wide range of sequential decision making problems.
Efficient RL algorithm for multinomial logistic MDPs with provable guarantees.
problem Model-based RL for episodic MDPs with unknown transition probabilities.
method Upper confidence bound-based algorithm for exploration-exploitation balance.
result Achieves regret bound for multinomial logistic models.
The paper improves model-based reinforcement learning by using multi-timestep objectives.
problem Compounding errors in one-step dynamics models as trajectory length increases.
method Developed a multi-timestep objective as a weighted sum of losses at various future horizons.
result Exponentially decaying weights significantly improve long-horizon performance.