The paper evaluates various bonus-based exploration methods in the ALE and finds limited improvement in performance.
problem Improving exploration in reinforcement learning algorithms, especially in challenging games.
method Empirical evaluation of different reward bonuses on the Arcade Learning Environment.
result Recently developed bonus-based exploration methods do not significantly improve performance in challenging games.
New exploration bonuses improve reinforcement learning efficiency.
problem Efficient exploration in unknown environments with limited feedback.
method Improved exploration bonuses scaling with 1/n and improved stopping time analysis.
result Faster learning rates and improved sample complexity in pure-exploration settings.
HyperX uses reward bonuses to enable efficient exploration in meta-learning.
problem Catastrophic failure of meta-learning with sparse rewards.
method HyperX uses novel reward bonuses to explore in approximate hyper-state space.
result HyperX meta-learns better task-exploration and adapts more successfully to new tasks.
Improved exploration in adversarial MDPs via dilated bonuses.
problem Global exploration in adversarial MDPs with local-search policies.
method Dilated bonuses to facilitate global exploration.
result Improved regret bounds in various MDP settings.
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space.…
The paper explores how state abstraction affects pseudo-count-based exploration bonuses.
problem Understanding the interplay between exploration and approximation in reinforcement learning.
method Quantifying MBIE-EB performance, relating density models to abstractions, and deriving new pseudo-count bonuses.
result A new pseudo-count bonus alleviates the mismatch between implicit and explicit abstractions.
FICM uses motion features to encourage exploration in DRL environments.
problem Exploiting novelty in environments with moving objects.
method Flow-based intrinsic curiosity module (FICM) leveraging motion features between consecutive frames.
result FICM encourages exploration in environments featuring moving objects.
New algorithms improve exploration in MDPs with theoretical guarantees.
problem Efficient exploration in undiscounted MDPs with continuous states.
method Exploration bonuses for SCAL and C-SCAL algorithms.
result Achieves sublinear regret with improved computational efficiency.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
Paper proposes a method for learning and planning in time-varying environments.
problem Learning and planning in unknown, time-varying environments.
method Computes the maximally likely model of the environment using maximum likelihood estimation.
result Generalizes learning algorithms for time-invariant Markov decision processes to time-varying ones.
Langevin DQN achieves deep exploration using Gaussian noise.
problem Deep exploration in reinforcement learning.
method Developed Langevin DQN, a variation of DQN with Gaussian noise.
result Langevin DQN achieves deep exploration.
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
Improves exploration in reinforcement learning with a latent dynamics model.
problem Handling the exploration-exploitation trade-off in reinforcement learning.
method Uses a latent dynamics model to derive reward bonuses as intrinsic motivation.
result Efficient exploration in high-dimensional state spaces, as demonstrated on continuous control tasks.
New algorithm optimizes exploration in unknown MDPs.
problem Optimal exploration in unknown Markov decision processes.
method Trajectory-optimization-based approximate solution using Bayesian experimental design.
result Empirically superior to intrinsic-motivation-based algorithms in convergence and model fidelity.
This paper extends the Bellman equation for reinforcement learning to explore uncertainty.
problem Exploration in reinforcement learning, focusing on uncertainty.
method Uncertainty Bellman Equation (UBE) to connect uncertainty at any time-step to expected uncertainties at subsequent time-steps.
result UBE's unique fixed point provides an upper bound on Q-values variance, improving DQN performance.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
New algorithms achieve logarithmic regret in KL-regularized Markov games.
problem Improving sample efficiency in game-theoretic settings with KL regularization.
method Developed OMG and SOMG algorithms for matrix and Markov games, using best response sampling and superoptimistic bonuses.
result Logarithmic regret in T T T that scales inversely with KL regularization strength β β β . New approach incentivizes strategic agents to explore, making exploration almost free.
problem Incentivized exploration in multi-armed bandits with long-term strategic agents.
method Simple incentive-provision strategy, best arm identification algorithm, and UCB lower bound.
result Exploration can be (almost) free when there are many learning agents.
RP1 uses active learning to improve world model in fewest samples.
problem Improving sample efficiency in MBRL for continuous control tasks.
method RP1 views MBRL as active learning, using a hybrid objective function and principled termination.
result Statistically significant gains over existing approaches on continuous control tasks.
Paper simplifies complex AI exploration by predicting future rewards.
problem Training machines to optimally gather complex information.
method Developed a denser reward structure using cross-value to decouple exploration and exploitation.
result Demonstrated successful learning of challenging tasks without shaping or bonuses.
RandQL is a new model-free algorithm for MDPs with a novel learning rate randomization approach.
problem Minimizing regret in episodic MDPs with model-free methods.
method Randomized Q-learning with learning rate randomization.
result RandQL achieves optimal regret bounds in both tabular and metric state-action spaces.
This paper improves Q-learning bounds using reference-advantage decomposition.
problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.
method Develops a novel error decomposition framework to prove gap-dependent regret bounds.
result Establishes logarithmic gap-dependent regret bounds for Q-learning.
We consider the problem of provably optimal exploration in reinforcement learning for finite horizon MDPs. We show that an optimistic modification to value iteration achieves a regret bound of O ~ ( H S A T + H 2 S 2 A + H T ) \tilde{O}( \sqrt{HSAT} + H^2S^2A+H\sqrt{T}) O ~ ( H S A T + H 2 S 2 A + H T ) where H H H is the time horizon, S S S the number of states, A A A the number of action…
Curriculum learning speeds up agent learning in Minecraft, a complex visual domain.
problem Training agents to learn multiple tasks in a complex, visual domain.
method Learning-progress based curriculum and dynamic exploration bonuses.
result Curriculum learning improves agent performance in a complex reinforcement learning problem.
New algorithm for personalized healthcare with privacy guarantees.
problem Online exploration in reinforcement learning with differential privacy constraints.
method ε-JDP algorithm with privately released exploration bonuses and visitation statistics.
result Regret bound of O ( S A H 2 T + S 2 A H 3 / ε ) O(\sqrt{SAH^2T}+S^2AH^3/ε) O ( S A H 2 T + S 2 A H 3 / ε ) matching information-theoretic lower bound. Improved reinforcement learning algorithm with linear approximation for unknown dynamics.
problem Reinforcement learning with adversarial changing cost functions and bandit feedback.
method Combines mirror-descent and least squares policy evaluation in an auxiliary MDP.
result Obtains an O ~ ( K 6 / 7 ) \widetilde O(K^{6/7}) O ( K 6/7 ) regret bound, significantly improving over previous methods. New method improves RL in continuous spaces with kernel smoothing.
problem Sample efficiency and structural assumptions in classical RL.
method Kernel smoothing model-based approach with Bernstein-style exploration bonus.
result Achieves improved regret bound in finite-horizon settings.
New algorithm tackles multi-agent reinforcement learning with optimal convergence rate.
problem Multi-agent reinforcement learning with large state spaces and linear function approximations.
method Refined AVLPR framework with data-dependent pessimistic estimation and action-dependent bonuses.
result First algorithm with optimal O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 ) convergence rate and no poly( A max A_{\max} A m a x ) dependency. New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
Improved sample complexity for identifying best policies in risk-sensitive reinforcement learning.
problem Identifying approximately optimal policies in risk-sensitive reinforcement learning with exponential horizon dependence.
method Forward-model based algorithm with KL-based exploration bonuses adapted for entropic criterion, leveraging smoothness properties of exponential utility and a new stopping rule.
result Achieved sample complexity matching the lower bound, closing the gap between upper and lower bounds.
Bayes-UCBVI tackles reinforcement learning with a new upper confidence bound method.
problem Optimizing exploration in reinforcement learning without bonuses.
method Bayes-UCBVI uses a quantile of a Q-value function posterior as an upper confidence bound.
result Proves a regret bound of order O ~ ( H 3 S A T ) \widetilde{O}(\sqrt{H^3SAT}) O ( H 3 S A T ) for tabular reinforcement learning. Study differential privacy in multi-agent RL, achieving efficient and private learning.
problem Protecting sensitive data in multi-agent reinforcement learning.
method Extending DP definitions to two-player games, designing an efficient algorithm with privatized bonuses.
result Achieved trajectory-wise differential privacy in multi-agent RL, improving regret bounds.
Proposes a new risk model using stable laws to manage company-wide losses.
problem Managing aggregate risks and pricing policies in the presence of systematic risk.
method Develops a modified risk model using multivariate stable distributions to account for various risk phenomena.
result Computes the Tail Conditional Expectation of aggregate risks and corresponding allocations.
Improved privacy in RL with near-optimal regret bounds.
problem Privacy-preserving reinforcement learning in personalized decision-making systems.
method Differentially private algorithm based on LSVI-UCB++ with privacy-preserving techniques.
result Achieved a near-optimal regret bound of O(d * sqrt(H^3 * K) + H^(15/4) * d^(7/6) * K^(1/2) / ε).
M 3 ^3 3 RL trains a manager to infer worker minds and assign tasks for optimal collaboration.
problem Optimal coordination among self-interested agents with diverse preferences and skills.
method Mind-aware Multi-agent Management Reinforcement Learning (M^3RL) that infers worker minds and assigns tasks.
result Effective in modeling worker minds and achieving optimal ad-hoc teaming.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
A new multi-objective RL framework improves intrinsic exploration performance.
problem Sub-optimal exploration performance due to ad-hoc handling of intrinsic exploration.
method A multi-objective RL framework where both exploration and exploitation are optimized as separate objectives.
result EMU-Q method outperforms classic and other intrinsic RL methods on benchmarks.
ISL algorithm tackles deep exploration efficiently.
problem Deep exploration in reinforcement learning.
method Derives ISL algorithm by augmenting RL objective with a novel regularization term.
result Empirically shows state-of-the-art performance on deep-exploration benchmarks.
Study explores how to efficiently explore communities with limited budget.
problem Maximizing the number of members met with limited budget in community exploration.
method Systematic study from offline optimization to online learning, including greedy methods and upper confidence algorithms.
result Achieved logarithmic and constant regret bounds in online learning setting.
EBE enables efficient exploration in reinforcement learning.
problem Inefficient exploration in reinforcement learning.
method Entropy-based exploration (EBE) quantifies learning and adaptively explores unexplored regions.
result EBE enables faster learning without hyperparameter tuning.
This work learns latent representations to speed up exploration in complex environments.
problem Challenging exploration in high-dimensional state and action spaces with sparse rewards.
method Representation learning using prior experience to learn effective latent representations.
result Learned latent representations reduce the dimensionality of the search space for effective exploration.
Go-Explore improves performance on hard-exploration problems in Atari games.
problem Challenges in reinforcement learning, especially with sparse or deceptive rewards.
method Exploits principles of remembering states, returning to promising states, and solving simulated environments.
result Scores significantly higher than previous state-of-the-art on Montezuma's Revenge and Pitfall.
This work tackles the exploration-exploitation dilemma in RL by developing optimal policies that are inherently exploration-conscious.
problem The exploration-exploitation tradeoff in Reinforcement Learning, where policies need to balance new action exploration with past experience exploitation.
method Developed exploration-conscious criteria that result in optimal policies, solving these criteria by solving a surrogate Markov Decision Process.
result Demonstrated superior performance of exploration-conscious RL algorithms compared to non-exploration-conscious counterparts in both discrete and continuous action spaces.
R3L uses planning algorithms to efficiently explore sparse reward environments.
problem Balancing exploration and exploitation in sparse reward reinforcement learning.
method Formulate exploration as a search problem using RRT, leverage demonstrations from initial solutions to refine RL policy.
result R3L outperforms classic and intrinsic exploration techniques, requiring fewer samples and achieving better asymptotic performance.
This work proposes sample complexity bounds for Q-learning with random exploration.
problem Understanding the sample efficiency of simple exploration strategies in reinforcement learning.
method Problem-specific sample complexity bounds for Q-learning with random walk exploration.
result Proposes bounds that relate to empirical performance in benchmark domains.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.