Improved exploration methods for reinforcement learning with reduced sample complexity.
problem Challenges in reinforcement learning exploration in unknown environments.
method Proposed game-theoretic and trajectory entropy algorithms with improved sample complexity.
result Established statistical advantage of entropy-regularized MDPs for exploration and reduced sample complexity.
Backward exploration reduces sample complexity in policy evaluation.
problem Empirical policy evaluation in reinforcement learning.
method Backward exploration algorithms from high-cost states.
result Reduced average-case sample complexity to O(logS). Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy explorat…
Study on sample complexity for pure exploration in feedback graph settings.
problem Sample complexity of pure exploration in online learning with feedback graphs.
method Derive instance-specific lower bounds and present asymptotically optimal algorithm TaS-FG.
result TaS-FG is asymptotically optimal and efficient across different graph configurations.
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space.…
Safe exploration in RF-RL doesn't increase sample complexity.
problem Achieving optimal policies with safety constraints in reward-free RL.
method Proposed SWEET framework for tabular and low-rank MDP settings, leveraging truncated value functions.
result Sample complexities match or outperform constraint-free counterparts, proving safety constraints have little impact.
Black-box optimizers that explore in parameter space have often been shown to outperform more sophisticated action space exploration methods developed specifically for the reinforcement learning problem. We examine these black-box methods closely to identify situations in which they are worse than action space explorat…
Efficient exploration is one of the key challenges for reinforcement learning (RL) algorithms. Most traditional sample efficiency bounds require strategic exploration. Recently many deep RL algorithms with simple heuristic exploration strategies that have few formal guarantees, achieve surprising success in many domain…
Balancing exploration and exploitation remains a key challenge in reinforcement learning (RL). State-of-the-art RL algorithms suffer from high sample complexity, particularly in the sparse reward case, where they can do no better than to explore in all directions until the first positive rewards are found. To mitigate …
The study analyzes batched methods for early stopping in stochastic multi-armed bandits.
problem Early stopping in stochastic multi-armed bandits with fixed confidence.
method Instance-dependent lower bounds and a general batched algorithm with upper bounds.
result Upper and lower bounds on the number of batches and sample complexity.
AceIRL learns expert reward from active exploration.
problem Learning reward function from expert demonstrations in unknown environments.
method Active exploration to infer reward function and identify good policy.
result First approach to active IRL with sample-complexity bounds that doesn't require a generative model.
The paper explores conditions for topological rigidity in quotients of the Davis complex.
problem Understanding when quotients of the Davis complex are topologically rigid.
method Analyzing quotients of the Davis complex of right-angled Coxeter groups and conditions on defining graphs.
result Introduction of infinitely many infinite topologically rigid subclasses.
New approach for reward-free exploration reduces estimation error.
problem Reward-free exploration in reinforcement learning.
method Adaptive approach reducing MDP estimation error.
result Reward-free UCRL algorithm improves sample complexity.
New framework uses unsupervised learning for efficient exploration in RL.
problem Efficient exploration in reinforcement learning with rich observations.
method Combines unsupervised learning and no-regret RL algorithms.
result Proves sample complexity for finding near-optimal policies.
Adaptive exploration scheme for evaluating multiple policies with different rewards.
problem Online multi-reward multi-policy evaluation.
method Adapted (ε,δ)-PAC perspective and MR-NaS exploration scheme to minimize sample complexity. result Demonstrated effectiveness of adaptive exploration in tabular domains.
Study task-guided exploration in linear dynamical systems, improving sample complexity.
problem Efficiently learning about an environment to complete a specific task.
method Proposed a computationally efficient experiment-design based exploration algorithm.
result Optimally explores the environment, collecting precise information needed to complete the task.
Simpler ε-greedy with longer action durations improves exploration.
problem Limited exploration capability of ε-greedy in complex domains.
method Temporally extended ε-greedy with repeated actions for random durations.
result Temporally extended ε-greedy outperforms sophisticated methods on various domains.
Improves RLHF sample efficiency by scaling reward complexity polynomially.
problem Exponential sample complexity in RLHF algorithms for skewed preferences.
method SE-POPO, an online RLHF algorithm that achieves polynomial sample complexity.
result SE-POPO outperforms existing algorithms in sample efficiency.
New exploration bonuses improve reinforcement learning efficiency.
problem Efficient exploration in unknown environments with limited feedback.
method Improved exploration bonuses scaling with 1/n and improved stopping time analysis.
result Faster learning rates and improved sample complexity in pure-exploration settings.
Paper proposes Adaptive Pareto Exploration for identifying Pareto optimal arms in multi-objective scenarios.
problem Identifying Pareto optimal arms in multi-objective scenarios with relaxed constraints.
method Adaptive Pareto Exploration strategy for different relaxations of Pareto Set Identification.
result Reduction in sample complexity when identifying at most k Pareto optimal arms.
A new strategy for identifying the best arm in Gaussian bandits with improved exploration.
problem Best-arm identification for Gaussian bandits with bounded means and unit variance.
method Exploration-Biased Sampling, a non-asymptotic approach with improved exploration behavior.
result Improved exploration behavior makes the strategy more stable and interpretable.
New algorithm achieves near optimal sample complexity for 1-identification problem.
problem Determining if an arm's mean reward is at least a known threshold with high probability.
method Design of Sequential-Exploration-Exploitation (SEE) algorithm with non-asymptotic analysis.
result Achieves near optimality in sample complexity, matching upper and lower bounds up to a polynomial logarithmic factor.
Task-agnostic RL tackles exploration in MDPs with multiple tasks.
problem Challenges in reinforcement learning with multiple tasks or conflicting objectives.
method Task-agnostic RL framework, UCBZero algorithm.
result UCBZero finds near-optimal policies for multiple tasks efficiently.
Survey explores interactions between convex and complex geometry.
problem Understanding intersections between convex and complex geometry.
method Survey and review of existing literature.
result Demonstrates fascinating interactions between convex and complex geometry.
Curriculum learning speeds up agent learning in Minecraft, a complex visual domain.
problem Training agents to learn multiple tasks in a complex, visual domain.
method Learning-progress based curriculum and dynamic exploration bonuses.
result Curriculum learning improves agent performance in a complex reinforcement learning problem.
Develops first optimal algorithm for logistic bandits.
problem Pure exploration in logistic bandits.
method Logistic track-and-stop (Log-TS) algorithm.
result Asymptotically matches lower bound for expected sample complexity.
Adaptive algorithm for outlier detection by balancing arm exploration and threshold estimation.
problem Identifying outliers in a set of rewards where the threshold is a function of all rewards.
method Adaptively updated confidence interval for the threshold based on previous rounds' estimates, balancing exploration of individual arms and the outlier threshold.
result Efficient algorithm with reduced sample complexity for outlier detection.
We determine the sample complexity of pure exploration bandit problems with multiple good answers. We derive a lower bound using a new game equilibrium argument. We show how continuity and convexity properties of single-answer problems ensures that the Track-and-Stop algorithm has asymptotically optimal sample complexi…
New findings on when to use action space exploration in reinforcement learning.
problem Understanding when to use action space exploration over traditional methods.
method Theoretical analysis and empirical testing of simple exploration methods.
result Exploration in action space is preferred when parametric complexity exceeds action space dimensionality and horizon length.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
DETC algorithm achieves asymptotic optimality in multi-armed bandit problems.
problem Multi-armed bandit problem with subgaussian rewards.
method DETC algorithm with two exploration and exploitation phases.
result DETC achieves asymptotically optimal regret bound.
Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often th…
New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of O(d2ε−2) for finding an ε-optimal policy. Balancing exploration and exploitation is a fundamental part of reinforcement learning, yet most state-of-the-art algorithms use a naive exploration protocol like ε-greedy. This contributes to the problem of high sample complexity, as the algorithm wastes effort by repeatedly visiting parts of the state space that ha…
New methods for efficient exploration under unknown linear constraints in bandits.
problem Optimizing decisions under unknown linear constraints in bandit problems.
method Lagrangian relaxation, computationally efficient extensions of existing methods, constraint-adaptive stopping rule.
result LAGEX achieves asymptotically optimal sample complexity, LATS shows asymptotic optimality up to novel constants.
An agent explores indefinitely in an environment with unlimited rewards.
problem Balancing exploration and exploitation in environments with unlimited rewards.
method Simple example of an environment with unbounded rewards and optimal agent behavior.
result An optimal agent always explores to maximize rewards, regardless of accumulated knowledge.
Latent variable models improve RL by facilitating efficient learning and exploration.
problem Improving sample efficiency in reinforcement learning.
method Representation view of latent variable models for state-action value functions, incorporating kernel embeddings and UCB exploration.
result Established sample complexity of the proposed approach in online and offline settings, demonstrated superior performance in benchmarks.
A new framework isolates exploration challenges in RL without explicit rewards.
problem Challenges in reinforcement learning, especially exploration.
method Reward-free RL framework, collecting trajectories without a reward function, then computing policies for various reward functions.
result Efficient algorithm that conducts exploration and computes near-optimal policies for multiple reward functions.
Paper simplifies complex AI exploration by predicting future rewards.
problem Training machines to optimally gather complex information.
method Developed a denser reward structure using cross-value to decouple exploration and exploitation.
result Demonstrated successful learning of challenging tasks without shaping or bonuses.
Survey explores cohomology's roles in applied math and sciences.
problem Understanding cohomology's role in solving differential equations.
method Examining differential complexes and structure-preserving discretizations.
result Various fundamental concepts in mechanics are formulated using differential complexes.
Improved exploration algorithm for unknown MDPs with reduced sample complexity.
problem Exploration of unknown environments without reward function.
method Incremental model-based approach that interleaves state discovery and policy improvement.
result Achieves sample complexity scaling as O~(L5SL+εΓL+εAε−2). This paper explores Khovanov adequacy in knot theory.
problem Understanding Khovanov homology and its adequacy.
method Using independence complexes and homotopy type calculations.
result Khovanov adequacy is explored within the context of independence complexes and homotopy type of extreme spectra.
AGAC uses an adversary to enhance exploration in complex tasks.
problem Sample inefficiency in complex environments, especially in tasks requiring efficient exploration.
method Integrates an adversary into the actor-critic framework to encourage innovative exploration strategies.
result AGAC leads to more exhaustive exploration and outperforms state-of-the-art methods.
Langevin DQN achieves deep exploration using Gaussian noise.
problem Deep exploration in reinforcement learning.
method Developed Langevin DQN, a variation of DQN with Gaussian noise.
result Langevin DQN achieves deep exploration.
Unified algorithm for efficient pure exploration using dual variables.
problem Efficiently achieving a specific goal through adaptive experimentation.
method Introducing dual variables to derive optimal allocation conditions, leading to Information-Directed Selection.
result Top-two Thompson sampling attains asymptotic optimality for Gaussian best-arm identification.
Using reinforcement learning to learn control policies is a challenge when the task is complex with potentially long horizons. Ensuring adequate but safe exploration is also crucial for controlling physical systems. In this paper, we use temporal logic to facilitate specification and learning of complex tasks. We combi…
E4 algorithm optimizes batched linear bandits with minimal regret and batches.
problem Optimizing batched linear bandits for minimal regret.
method Explore-Estimate-Eliminate-Exploit framework with optimal exploration rate.
result Achieves minimax and asymptotic optimality in regret and batch complexity.
Building agents that can explore their environments intelligently is a challenging open problem. In this paper, we make a step towards understanding how a hierarchical design of the agent's policy can affect its exploration capabilities. First, we design EscapeRoom environments, where the agent must figure out how to n…