BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
A new multi-objective RL framework improves intrinsic exploration performance.
problem Sub-optimal exploration performance due to ad-hoc handling of intrinsic exploration.
method A multi-objective RL framework where both exploration and exploitation are optimized as separate objectives.
result EMU-Q method outperforms classic and other intrinsic RL methods on benchmarks.
ISL algorithm tackles deep exploration efficiently.
problem Deep exploration in reinforcement learning.
method Derives ISL algorithm by augmenting RL objective with a novel regularization term.
result Empirically shows state-of-the-art performance on deep-exploration benchmarks.
Study explores how to efficiently explore communities with limited budget.
problem Maximizing the number of members met with limited budget in community exploration.
method Systematic study from offline optimization to online learning, including greedy methods and upper confidence algorithms.
result Achieved logarithmic and constant regret bounds in online learning setting.
EBE enables efficient exploration in reinforcement learning.
problem Inefficient exploration in reinforcement learning.
method Entropy-based exploration (EBE) quantifies learning and adaptively explores unexplored regions.
result EBE enables faster learning without hyperparameter tuning.
This work learns latent representations to speed up exploration in complex environments.
problem Challenging exploration in high-dimensional state and action spaces with sparse rewards.
method Representation learning using prior experience to learn effective latent representations.
result Learned latent representations reduce the dimensionality of the search space for effective exploration.
Go-Explore improves performance on hard-exploration problems in Atari games.
problem Challenges in reinforcement learning, especially with sparse or deceptive rewards.
method Exploits principles of remembering states, returning to promising states, and solving simulated environments.
result Scores significantly higher than previous state-of-the-art on Montezuma's Revenge and Pitfall.
This work tackles the exploration-exploitation dilemma in RL by developing optimal policies that are inherently exploration-conscious.
problem The exploration-exploitation tradeoff in Reinforcement Learning, where policies need to balance new action exploration with past experience exploitation.
method Developed exploration-conscious criteria that result in optimal policies, solving these criteria by solving a surrogate Markov Decision Process.
result Demonstrated superior performance of exploration-conscious RL algorithms compared to non-exploration-conscious counterparts in both discrete and continuous action spaces.
R3L uses planning algorithms to efficiently explore sparse reward environments.
problem Balancing exploration and exploitation in sparse reward reinforcement learning.
method Formulate exploration as a search problem using RRT, leverage demonstrations from initial solutions to refine RL policy.
result R3L outperforms classic and intrinsic exploration techniques, requiring fewer samples and achieving better asymptotic performance.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.
New exploration bonuses improve reinforcement learning efficiency.
problem Efficient exploration in unknown environments with limited feedback.
method Improved exploration bonuses scaling with 1/n and improved stopping time analysis.
result Faster learning rates and improved sample complexity in pure-exploration settings.
New findings on when to use action space exploration in reinforcement learning.
problem Understanding when to use action space exploration over traditional methods.
method Theoretical analysis and empirical testing of simple exploration methods.
result Exploration in action space is preferred when parametric complexity exceeds action space dimensionality and horizon length.
Proposes a method to avoid excessive exploration in reinforcement learning.
problem Avoiding excessive exploration in reinforcement learning to deploy it in practice.
method Designs a novel algorithm using UCB reinforcement learning policy with adaptive exploration constraints.
result Proves that the approach remains conservative while minimizing regret in tabular settings and validates on real-world tasks.
New algorithm reduces regret by allowing free exploration in multi-armed bandits.
problem Designing an adaptive policy to minimize regret with a free exploration budget.
method Introduced ( α , β ) (α,β) ( α , β ) -probably saving policies and a two-phase algorithm UFE-KLUCB-H. result UFE-KLUCB-H accumulates strictly less regret than non-free exploration policies.
New meta-RL method avoids exploration-exploitation trade-off.
problem Learning to explore and exploit simultaneously in meta-RL.
method Developed new objectives for exploration and exploitation.
result DREAM outperforms existing methods on complex tasks.
The paper tackles pure exploration in multi-armed bandits with low rank structure using oblivious sampling.
problem Pure exploration in multi-armed bandits with low rank reward sequences.
method The approach involves separating the exploration strategy from feedback, using oblivious sampling, and incorporating kernel information of reward vectors.
result Efficient algorithms with regret bound O ( d ( ln N ) / n ) O(d\sqrt{(\ln N)/n}) O ( d ( ln N ) / n ) for both time-varying and fixed cases, with a lower bound gap of O ( ln N ) O(\sqrt{\ln N}) O ( ln N ) . The paper evaluates various bonus-based exploration methods in the ALE and finds limited improvement in performance.
problem Improving exploration in reinforcement learning algorithms, especially in challenging games.
method Empirical evaluation of different reward bonuses on the Arcade Learning Environment.
result Recently developed bonus-based exploration methods do not significantly improve performance in challenging games.
MAME models a separate exploration policy for faster adaptation.
problem Efficient exploration strategies for quick task adaptation in meta-reinforcement learning.
method Explicitly models a separate exploration policy for task distribution, using self-supervised or supervised learning objectives for adaptation.
result Superior performance compared to prior works in meta-reinforcement learning.
Regularization-induced exploration improves contextual bandit performance.
problem Complex reward models in real-world contextual bandits are hard to explore effectively.
method Regularization-induced exploration using stochasticity in cross-validation.
result Regularization-induced exploration leads to reliable exploration in large-scale business environments.
The paper proposes an algorithm to learn efficient and effective exploration policies in reinforcement learning.
problem Balancing exploration and exploitation in reinforcement learning.
method Formalized a counterfactual metric for exploration utility and used meta-learning to learn an end-to-end exploration policy.
result Demonstrated improved performance in high-dimensional control tasks in MuJoCo simulator compared to previous methods.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
Efficient exploration is one of the key challenges for reinforcement learning (RL) algorithms. Most traditional sample efficiency bounds require strategic exploration. Recently many deep RL algorithms with simple heuristic exploration strategies that have few formal guarantees, achieve surprising success in many domain…
Maximizes Rényi entropy for efficient exploration in reward-free RL.
problem Challenges of exploration in reward-free reinforcement learning.
method Maximizes Rényi entropy over state-action space in exploration phase; uses batch RL for planning phase.
result Effective and sample-efficient exploration leading to superior policies.
The paper explores how state abstraction affects pseudo-count-based exploration bonuses.
problem Understanding the interplay between exploration and approximation in reinforcement learning.
method Quantifying MBIE-EB performance, relating density models to abstractions, and deriving new pseudo-count bonuses.
result A new pseudo-count bonus alleviates the mismatch between implicit and explicit abstractions.
An agent explores indefinitely in an environment with unlimited rewards.
problem Balancing exploration and exploitation in environments with unlimited rewards.
method Simple example of an environment with unbounded rewards and optimal agent behavior.
result An optimal agent always explores to maximize rewards, regardless of accumulated knowledge.
Proposes EVE for efficient exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning.
method EVE: a recipe for posterior over parameters, facilitating efficient exploration.
result Competitive performance on benchmarks, efficient exploration confirmed.
New DRL algorithm improves sample efficiency and exploration performance.
problem High sample cost in deep reinforcement learning.
method Combines entropy and bootstrap techniques with Tsallis entropy regularization.
result Demonstrates more efficient and effective exploration on Atari games.
A new exploration method for RL using parameter space noise.
problem Improving exploration in deep reinforcement learning.
method Switching isotropic and directional exploration in parameter space with parameter space noise.
result The proposed method achieves competitive results and better performance in sparse reward environments.
VASE uses Bayesian neural networks to improve exploration in sparse reward environments.
problem Exploration in environments with continuous control and sparse rewards.
method VASE uses a Bayesian neural network model of the environment dynamics and variational inference to alternately update the model's accuracy and policy.
result VASE outperforms other surprise-based exploration techniques in continuous control sparse reward environments.
ASE safely explores unknown MDPs with unknown dynamics, improving sample efficiency.
problem Balancing exploration and safety in unknown MDPs with stochastic dynamics.
method Exploits analogies between state-action pairs to safely learn near-optimal policies.
result Empirically improves sample efficiency compared to existing methods.
This work uses model uncertainty for efficient exploration in sparse reward environments.
problem Challenging exploration in sparse reward reinforcement learning environments.
method Implicit generative modeling approach to estimate Bayesian uncertainty of the agent's belief of the environment dynamics.
result Our implicit generative model consistently outperforms competing approaches in data efficiency for exploration.
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space.…
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
Two new exploration methods for multi-agent systems improve team performance.
problem Exploration in transition-dependent multi-agent settings.
method EITI and EDTI, using mutual information and VoI to encourage coordinated exploration.
result Significant improvement in multi-agent performance through coordinated exploration.
EMI uses predictive signals to guide exploration in sparse reward settings.
problem Challenges of reinforcement learning with sparse reward signals.
method Constructs embedding representations of states and actions for forward prediction in the representation space.
result Competitive results on challenging tasks with continuous control and discrete actions.
AdaLinUCB optimizes exploration-exploitation for contextually varying costs.
problem Optimizing decision-making in environments with varying exploration costs.
method Adaptive Upper-Confidence-Bound (AdaLinUCB) algorithm for opportunistic learning.
result AdaLinUCB achieves O((log T)^2) regret bound, significantly outperforming other algorithms.
This paper proposes a self-supervised exploration method using disagreement of dynamics models.
problem Efficient exploration in stochastic environments with real robots.
method Train an ensemble of dynamics models and incentivize exploration to maximize disagreement.
result Sample-efficient exploration achieved without external rewards or reinforcement learning.
MAX uses ensemble models to efficiently explore novel situations in reinforcement learning.
problem Efficient exploration in reinforcement learning.
method Model-based active exploration using ensemble forward models and Bayesian disagreement.
result MAX is at least an order of magnitude more efficient than baselines in semi-random environments.
New RL method finds near-optimal policies efficiently.
problem Efficient exploration in model-free RL.
method Modifies Delayed Q-learning with long-term exploration bonus.
result Polynomial time PAC-MDP solution for near-optimal policy finding.
Simple method improves exploration in various decision problems.
problem Improving exploration in sequential decision problems.
method Parameterized Exploration (PE) method that considers time horizon and state of knowledge.
result PE outperforms un-tuned methods in various bandit and decision problem settings.
Proposes a method to enhance exploration in RL using temporal difference uncertainties.
problem Challenges in estimating uncertainty in non-tabular reinforcement learning settings.
method Estimates uncertainty over value function using temporal difference errors and incorporates it as an intrinsic reward.
result Demonstrates improved exploration in various tasks, including Deep Sea and Atari 2600 environments.
Proposes EE-Net for neural exploration in contextual bandits.
problem Exploitation-Exploration tradeoff in contextual bandits.
method Uses two neural networks: Exploitation and Exploration, to learn reward function and adaptively explore.
result Achieves O ( T log T ) \mathcal{O}(\sqrt{T\log T}) O ( T log T ) regret and outperforms existing methods. Task-agnostic RL tackles exploration in MDPs with multiple tasks.
problem Challenges in reinforcement learning with multiple tasks or conflicting objectives.
method Task-agnostic RL framework, UCBZero algorithm.
result UCBZero finds near-optimal policies for multiple tasks efficiently.
Hyper addresses the hyperparameter tuning challenge in RL, improving exploration efficiency and robustness.
problem Hyperparameter tuning is a significant challenge in RL, especially for curiosity-based exploration methods.
method Hyper robustly explores by effectively regularizing exploration visits and decoupling exploitation.
result Hyper is provably efficient and robust in various RL environments.
Algorithm achieves optimal pricing with minimal exploration for dynamic markets.
problem Optimal pricing in dynamic markets with contextual information.
method Localized exploration-then-commit (LetC) algorithm with pure exploration, refinement, and exploitation stages.
result Achieves minimax optimal, dimension-free regret bound.
Langevin DQN achieves deep exploration using Gaussian noise.
problem Deep exploration in reinforcement learning.
method Developed Langevin DQN, a variation of DQN with Gaussian noise.
result Langevin DQN achieves deep exploration.
New L 1 L_1 L 1 -Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces L 1 L_1 L 1 -Coverage objective to enable efficient exploration and planning. result First computationally efficient algorithms for online reinforcement learning with low coverability.