This work learns latent representations to speed up exploration in complex environments.
problem Challenging exploration in high-dimensional state and action spaces with sparse rewards.
method Representation learning using prior experience to learn effective latent representations.
result Learned latent representations reduce the dimensionality of the search space for effective exploration.
Recently deep reinforcement learning (DRL) has achieved outstanding success on solving many difficult and large-scale RL problems. However the high sample cost required for effective learning often makes DRL unaffordable in resource-limited applications. With the aim of improving sample efficiency and learning performa…
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
The paper proposes an algorithm to learn efficient and effective exploration policies in reinforcement learning.
problem Balancing exploration and exploitation in reinforcement learning.
method Formalized a counterfactual metric for exploration utility and used meta-learning to learn an end-to-end exploration policy.
result Demonstrated improved performance in high-dimensional control tasks in MuJoCo simulator compared to previous methods.
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
InfoBot learns decision states from prior experience to guide exploration.
problem Discovering effective policies in sparse reward environments.
method InfoBot uses an information bottleneck to learn decision states from prior experience.
result InfoBot effectively identifies decision states, even in partially observed settings.
SU improves exploration in reinforcement learning, surpassing human performance on Atari games.
problem Challenges in scaling PSRL for reinforcement learning with neural networks.
method Design and implementation of Successor Uncertainties (SU) algorithm.
result SU outperforms human performance on Atari games and surpasses RVF competitor Bootstrapped DQN.
Uncovering how college effects vary by subgroups using machine learning.
problem Exploring how college effects vary by subgroups in observational data.
method Causal trees for recursive subgrouping, honest estimation, leaf-specific matching, sensitivity analyses.
result Identifies new subgroups of individuals whose college effects on wages vary significantly.
REN addresses uncertainty in user feedbacks for better recommendation systems.
problem Recurrent neural networks focus solely on item relevance, neglecting diverse item exploration.
method Proposes REN, a new type of recurrent neural network that balances relevance and exploration while accounting for representation uncertainty.
result REN achieves satisfactory long-term rewards on synthetic and real-world recommendation datasets, outperforming state-of-the-art models.
Meta-agent learns effective exploration from offline data.
problem Design a meta-agent to quickly maximize reward in unseen tasks.
method Bayesian RL approach with adaptive neural belief estimate.
result Meta-agent learns effective exploration behavior from diverse tasks.
Regularization-induced exploration improves contextual bandit performance.
problem Complex reward models in real-world contextual bandits are hard to explore effectively.
method Regularization-induced exploration using stochasticity in cross-validation.
result Regularization-induced exploration leads to reliable exploration in large-scale business environments.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.
CAE repurposes critics for efficient exploration in RL.
problem Challenges in reinforcement learning, especially exploration.
method Repurposing value networks in RL algorithms for exploration.
result Provably sub-linear regret bounds and strong empirical stability.
We show how an ensemble of Q∗-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the Q-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experimen…
A method to train multi-agent reinforcement learning models without intrinsic rewards.
problem Training multi-agent reinforcement learning models with sparse rewards is challenging.
method A learning-based exploration strategy using variational graph autoencoder to generate initial states.
result The method improves the training and performance of multi-agent reinforcement learning models.
Bayesian optimization offers the possibility of optimizing black-box operations not accessible through traditional techniques. The success of Bayesian optimization methods such as Expected Improvement (EI) are significantly affected by the degree of trade-off between exploration and exploitation. Too much exploration c…
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
New approach uses unlabeled prior data to accelerate exploration in sparse reward tasks.
problem Sparse reward tasks in reinforcement learning.
method Learn reward model from online experience, label prior data, and use concurrently.
result Rapid exploration in challenging sparse-reward domains.
Incentivized exploration improves MAB performance under reward drift.
problem Improving exploration in multi-armed bandits with biased feedback.
method Analysis of three MAB algorithms (UCB, ε-Greedy, Thompson Sampling) under drifted reward feedback.
result All algorithms achieve O(logT) regret and compensation under drifted reward. ADAC uses analogous policies to improve RL exploration without sacrificing stability.
problem Improving RL exploration without compromising stability and expressiveness.
method Disentangled actor-critic approach with analogous pairs of actors and critics.
result Empirical evaluation shows ADAC outperforms alternatives in challenging exploration tasks.
Simple method improves exploration in various decision problems.
problem Improving exploration in sequential decision problems.
method Parameterized Exploration (PE) method that considers time horizon and state of knowledge.
result PE outperforms un-tuned methods in various bandit and decision problem settings.
EQO uses a simple bonus term for efficient exploration in tabular RL.
problem Achieving minimax optimal reinforcement learning with practical efficiency.
method Exploration via Quasi-Optimism, employing a bonus proportional to state-action visit count.
result Achieves the sharpest known regret bound for tabular RL.
EBE enables efficient exploration in reinforcement learning.
problem Inefficient exploration in reinforcement learning.
method Entropy-based exploration (EBE) quantifies learning and adaptively explores unexplored regions.
result EBE enables faster learning without hyperparameter tuning.
Proposes meTS for efficient exploration in correlated bandits.
problem Efficiently explore in bandits with correlated actions.
method Mixed-effect model with Thompson Sampling.
result Bound on Bayes regret with two terms.
This paper presents the Homeo-Heterostatic Value Gradients (HHVG) algorithm as a formal account on the constructive interplay between boredom and curiosity which gives rise to effective exploration and superior forward model learning. We envisaged actions as instrumental in agent's own epistemic disclosure. This motiva…
Hyper addresses the hyperparameter tuning challenge in RL, improving exploration efficiency and robustness.
problem Hyperparameter tuning is a significant challenge in RL, especially for curiosity-based exploration methods.
method Hyper robustly explores by effectively regularizing exploration visits and decoupling exploitation.
result Hyper is provably efficient and robust in various RL environments.
This paper improves DQN agents' robustness to adversarial perturbations.
problem Improving DQN agents' robustness to adversarial perturbations.
method Adversarial training and a novel AGE mechanism based on ε-greedy and Boltzmann exploration. result AGE mechanism enhances DQN agents' robustness to adversarial perturbations.
Deep reinforcement learning improves medical test suggestions.
problem Improving accuracy of disease diagnosis through better medical test recommendations.
method Formulated as a stage-wise Markov decision process, trained using reinforcement learning with a new action policy representation and exploration scheme.
result Significant improvement in disease diagnosis accuracy with better medical test suggestions.
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
Q-chunking improves RL for long tasks by chunking actions.
problem Improving sample efficiency and exploration in offline-to-online RL.
method Action chunking in TD-based RL methods.
result Q-chunking outperforms prior methods on long-horizon tasks.
We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of least-squares value iteration that use Boltzmann or epsilon-greedy exploration can be highly ineffic…
New algorithms improve deep RL with entropy-based action selection and environment exploration.
problem Improving deep reinforcement learning algorithms for better sample efficiency and effectiveness.
method Proposes Tsallis entropy Actor-Critic (TAC), Renyi entropy Actor-Critic (RAC), and Ensemble Actor-Critic (EAC) algorithms.
result Empirically, TAC, RAC, and EAC outperform SAC and other algorithms in benchmark control tasks.
Regularized policies are robust to adversarial rewards.
problem Understanding the effects of regularization on policy exploration and robustness.
method Using Fenchel duality to derive the dual problem of the regularized RL objective, showing the optimal policy is robust to adversarial rewards.
result Regularized policies are optimal for a reinforcement learning problem under adversarial reward conditions.
BeBold improves exploration in sparse-reward tasks by regulating visitation counts.
problem Efficient exploration in deep reinforcement learning under sparse rewards.
method Regulated difference of inverse visitation counts.
result BeBold solves 12 challenging tasks in MiniGrid with fewer steps than previous state-of-the-art.
A single policy suffices for near-optimal parallel exploration in RL.
problem Quantitative effects of parallel exploration in reward-free RL.
method Using a single policy to guide exploration across all agents.
result Near-linear speedup and near-minimax optimality for linear MDPs.
Thompson Sampling's performance degrades with approximate inference, especially under α-divergence.
problem Thompson Sampling's performance degradation due to approximate inference.
method Study of approximate inference effects on Thompson Sampling in k-armed bandit problems. result Small inference error can lead to poor performance (linear regret) in Thompson Sampling.
Paper proposes efficient online estimation of causal effects by deciding which data sources to query.
problem Data fusion problems with multiple data sources capturing distinct subsets of variables.
method Online moment selection (OMS) framework, balancing exploration and exploitation.
result OMS algorithms achieve zero asymptotic regret for estimating average treatment effects.
AutoFS combines trainers to improve feature selection efficiency and effectiveness.
problem Balancing feature selection efficiency and effectiveness.
method Interactive Reinforced Feature Selection (IRFS) framework with diverse trainers.
result Improved feature selection efficiency and effectiveness compared to existing methods.
Optimally explores dynamical systems with varying properties using context inference.
problem Learning dynamics models for systems with varying properties.
method Formulates dynamics models as stochastic processes conditioned on a latent context variable inferred from system transitions. Uses probabilistic formulation to compute optimal action sequences for exploration.
result Demonstrates effectiveness of the method on non-linear toy-problems and reinforcement learning environments.
Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristi…
New RL algorithm explains why deep learning works in stochastic environments.
problem Why deep RL algorithms perform well in practice despite using random exploration.
method Introducing SQIRL, an iterative RL algorithm that separates exploration and learning.
result Effective horizon explains why deep RL works in stochastic environments.
Improved exploration in RL with latent state marginalization.
problem Complexity of deep probabilistic models limits their practical use in reinforcement learning.
method Adopting latent variable policies within the MaxEnt framework, with low-cost marginalization of latent states.
result Effective marginalization leads to better exploration and more robust training.
Enhanced Gaussian process models accelerate optimization and posterior approximation.
problem Improving the accuracy and speed of Gaussian process models for optimization and inference.
method Introduces a random exploration step to classical GP-UCB algorithms, facilitating faster convergence.
result New algorithms achieve nearly optimal convergence rates and provide bounds for Hellinger distance.
Improves exploration in reinforcement learning with diverse population.
problem Lack of diversity in reinforcement learning agents leads to limited exploration.
method Optimizes a population of agents using volume of behavioral manifold and online learning techniques.
result Improves exploration without sacrificing performance.
Model-based compression is an effective, facilitating, and expanded model of neural network models with limited computing and low power. However, conventional models of compression techniques utilize crafted features [2,3,12] and explore specialized areas for exploration and design of large spaces in terms of size, spe…
Addressing the issue of SVMs parameters optimization, this study proposes an efficient memetic algorithm based on Particle Swarm Optimization algorithm (PSO) and Pattern Search (PS). In the proposed memetic algorithm, PSO is responsible for exploration of the search space and the detection of the potential regions with…
This technical note presents a new approach to carrying out the kind of exploration achieved by Thompson sampling, but without explicitly maintaining or sampling from posterior distributions. The approach is based on a bootstrap technique that uses a combination of observed and artificially generated data. The latter s…
The paper explores hyperparameters affecting adversarial robustness.
problem Understanding hyperparameters for adversarial robustness.
method Sensitivity analysis and hyperparameter optimization.
result Complex hyperparameter landscape affects adversarial robustness.