We address the challenge of effective exploration while maintaining good performance in policy gradient methods. As a solution, we propose diverse exploration (DE) via conjugate policies. DE learns and deploys a set of conjugate policies which can be conveniently generated as a byproduct of conjugate gradient descent. …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “exploration policies”
Study how untrained policies explore in RL environments.
problem Challenges in reinforcement learning, especially sparse or adversarial reward structures.
method Theoretical and empirical analysis of untrained deep neural policies in a toy model.
result Untrained policies generate correlated actions and non-trivial state-visitation distributions.
Adaptive exploration scheme for evaluating multiple policies with different rewards.
problem Online multi-reward multi-policy evaluation.
method Adapted -PAC perspective and MR-NaS exploration scheme to minimize sample complexity.
result Demonstrated effectiveness of adaptive exploration in tabular domains.
Revel tackles safe exploration in RL with verified symbolic policies.
problem Computational infeasibility of verifying neural networks in RL learning loops.
method Two policy classes: neurosymbolic with approximate gradients and symbolic policies for efficient verification. Mirror descent over policies to safely update and project policies.
result Revel discovers policies that outperform prior approaches to verified exploration.
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
Improved exploration in cooperative multi-agent reinforcement learning.
problem Limited expressiveness of Gaussian policies in DecSPG hinders effective exploration.
method Proposes decentralized diffusion policy learning (DDPL) with denoising diffusion probabilistic models.
result Consistently improved performance on various MARL benchmarks.
Proposes a conservative exploration method for RL agents.
problem Guaranteeing performance of exploratory policies in RL.
method Importance sampling for off-policy policy evaluation.
result Derives a regret bound ensuring no conservative constraint violation.
Maximizes Rényi entropy for efficient exploration in reward-free RL.
problem Challenges of exploration in reward-free reinforcement learning.
method Maximizes Rényi entropy over state-action space in exploration phase; uses batch RL for planning phase.
result Effective and sample-efficient exploration leading to superior policies.
A single policy suffices for near-optimal parallel exploration in RL.
problem Quantitative effects of parallel exploration in reward-free RL.
method Using a single policy to guide exploration across all agents.
result Near-linear speedup and near-minimax optimality for linear MDPs.
We study algorithms for average-cost reinforcement learning problems with value function approximation. Our starting point is the recently proposed POLITEX algorithm, a version of policy iteration where the policy produced in each iteration is near-optimal in hindsight for the sum of all past value function estimates. …
ADAC uses analogous policies to improve RL exploration without sacrificing stability.
problem Improving RL exploration without compromising stability and expressiveness.
method Disentangled actor-critic approach with analogous pairs of actors and critics.
result Empirical evaluation shows ADAC outperforms alternatives in challenging exploration tasks.
DAC enhances exploration in reinforcement learning with entropy regularization.
problem Improving exploration efficiency in reinforcement learning.
method Sample-aware entropy regularization using replay buffer action distributions.
result DAC significantly outperforms existing algorithms in reinforcement learning tasks.
A new method combines online and offline learning to tackle contextual bandits with missing action support.
problem Learning optimal policies with logged data when the logging policy has deficient support.
method Hybrid approach using online exploration to exploit supported actions and offline learning to avoid unnecessary explorations.
result Determines an optimal policy with theoretical guarantees using minimal online explorations.
This work learns exploration policies for unknown distributions using samples and policy gradients.
problem Learning exploration policies for unknown distributions in Bayesian bandits.
method Meta-learning approach parameterizing policies in a differentiable way and optimizing them with policy gradients.
result Effective gradient estimators and variance reduction techniques are derived.
New algorithm reduces regret by allowing free exploration in multi-armed bandits.
problem Designing an adaptive policy to minimize regret with a free exploration budget.
method Introduced -probably saving policies and a two-phase algorithm UFE-KLUCB-H.
result UFE-KLUCB-H accumulates strictly less regret than non-free exploration policies.
Adapts safe policies for exploration in high-risk settings.
problem Balancing safety and exploration in high-risk environments.
method Uses conformal calibration on a safe reference policy to determine aggressive action limits.
result Safe exploration improves performance without requiring model class identification or hyperparameter tuning.
Meta-Reinforcement learning approaches aim to develop learning procedures that can adapt quickly to a distribution of tasks with the help of a few examples. Developing efficient exploration strategies capable of finding the most useful samples becomes critical in such settings. Existing approaches towards finding effic…
Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous exploration methods either rely on complex structure to estimate the novelty of states, or incur sens…
OPPO optimizes policy learning with provable efficiency.
problem Designing efficient policy optimization algorithms that incorporate exploration.
method Proposes an Optimistic Proximal Policy Optimization (OPPO) algorithm.
result Achieves regret in episodic MDPs.
Paper proposes a policy-search algorithm to learn entropy-maximizing exploration policies in reward-free environments.
problem Reward-free learning in high-dimensional, continuous-control domains.
method Maximum Entropy POLicy optimization (MEPOL) algorithm that maximizes a non-parametric state entropy estimate.
result MEPOL learns a maximum-entropy exploration policy that facilitates learning various reward-based tasks.
Backward exploration reduces sample complexity in policy evaluation.
problem Empirical policy evaluation in reinforcement learning.
method Backward exploration algorithms from high-cost states.
result Reduced average-case sample complexity to .
RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.
problem Policy Optimization in continuous control tasks.
method RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for regret minimization in PO.
result Achieving constant regret under certain circumstances in PO with mediator feedback.
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…
A fundamental issue in reinforcement learning algorithms is the balance between exploration of the environment and exploitation of information already obtained by the agent. Especially, exploration has played a critical role for both efficiency and efficacy of the learning process. However, Existing works for explorati…
PC-PG balances exploration and exploitation in reinforcement learning.
problem Local policy gradient methods struggle with exploration.
method PC-PG uses an ensemble of learned policies (policy cover) to balance exploration and exploitation.
result PC-PG provides strong theoretical guarantees and empirical validation.
The Exploration-Exploitation tradeoff arises in Reinforcement Learning when one cannot tell if a policy is optimal. Then, there is a constant need to explore new actions instead of exploiting past experience. In practice, it is common to resolve the tradeoff by using a fixed exploration mechanism, such as -greedy ex…
What is a good exploration strategy for an agent that interacts with an environment in the absence of external rewards? Ideally, we would like to get a policy driving towards a uniform state-action visitation (highly exploring) in a minimum number of steps (fast mixing), in order to ease efficient learning of any goal-…
The ability to discover approximately optimal policies in domains with sparse rewards is crucial to applying reinforcement learning (RL) in many real-world scenarios. Approaches such as neural density models and continuous exploration (e.g., Go-Explore) have been proposed to maintain the high exploration rate necessary…
This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.
problem Lack of exploration in state space when maximizing policy entropy.
method Proposes maximizing the entropy of a lower bound approximation to the state weighting distribution, based on latent space representation.
result Entropy regularization based on marginal state distribution achieves superior state space coverage and better performance in various domains.
New -Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces -Coverage objective to enable efficient exploration and planning.
result First computationally efficient algorithms for online reinforcement learning with low coverability.
Regularized policies are robust to adversarial rewards.
problem Understanding the effects of regularization on policy exploration and robustness.
method Using Fenchel duality to derive the dual problem of the regularized RL objective, showing the optimal policy is robust to adversarial rewards.
result Regularized policies are optimal for a reinforcement learning problem under adversarial reward conditions.
A new framework isolates exploration challenges in RL without explicit rewards.
problem Challenges in reinforcement learning, especially exploration.
method Reward-free RL framework, collecting trajectories without a reward function, then computing policies for various reward functions.
result Efficient algorithm that conducts exploration and computes near-optimal policies for multiple reward functions.
ENIAC method optimizes and explores complex RL problems with non-linear policies.
problem Theoretical understanding of non-linear policies in RL with strategic exploration.
method ENIAC, an actor-critic method for non-linear function approximation.
result ENIAC finds near-optimal policies in polynomial exploration rounds under bounded eluder dimension.
Study on rapid policy changes in reinforcement learning.
problem Rapid change of greedy policy in reinforcement learning.
method Empirical study and ablation analysis.
result Policy churn is a beneficial form of implicit exploration.
DiCE uses diverse agents to explore and learn, avoiding local minima.
problem Local minima in RL due to limited exploration and correlated behavior.
method DiCE employs a group of heterogeneous agents to explore simultaneously and share experiences, with a diversity regularization mechanism.
result DiCE achieves substantial improvement over baselines in MuJoCo locomotion tasks.
New method learns adaptive exploration strategies for dynamic tasks.
problem Learning effective exploration strategies in changing environments.
method Informed policy regularization to reduce sample complexity of RNN-based policies.
result Method learns efficient exploration strategies balancing information gathering and reward maximization.
New method uses entropy to improve policy gradient exploration.
problem Limited exploration in policy gradient methods.
method Entropy regularization with discounted future state distribution.
result Proves convergence to locally optimal policy.
In recent years, state-of-the-art game-playing agents often involve policies that are trained in self-playing processes where Monte Carlo tree search (MCTS) algorithms and trained policies iteratively improve each other. The strongest results have been obtained when policies are trained to mimic the search behaviour of…
Agent learns directed exploration policies to improve performance in hard games.
problem Improving exploration in complex games.
method Episodic memory-based intrinsic reward, self-supervised inverse dynamics, UVFA framework.
result Doubles performance in hard exploration games, achieves non-zero rewards in Pitfall!.
This paper analyzes MORL and proposes efficient algorithms to learn Pareto optimal policies.
problem Understanding and efficiently learning Pareto optimal policies in multi-objective reinforcement learning.
method Systematic analysis of optimization targets, reformulation of Tchebycheff scalarization, online UCB-based algorithm, preference-free framework.
result Identification of Tchebycheff scalarization as a favorable method and efficient algorithms for learning Pareto optimal policies.
Reinforcement learning algorithms rely on exploration to discover new behaviors, which is typically achieved by following a stochastic policy. In continuous control tasks, policies with a Gaussian distribution have been widely adopted. Gaussian exploration however does not result in smooth trajectories that generally c…
Proposes a method to avoid excessive exploration in reinforcement learning.
problem Avoiding excessive exploration in reinforcement learning to deploy it in practice.
method Designs a novel algorithm using UCB reinforcement learning policy with adaptive exploration constraints.
result Proves that the approach remains conservative while minimizing regret in tabular settings and validates on real-world tasks.
New approach for reward-free exploration reduces estimation error.
problem Reward-free exploration in reinforcement learning.
method Adaptive approach reducing MDP estimation error.
result Reward-free UCRL algorithm improves sample complexity.
MAPS and MAPS-SE learn from multiple suboptimal experts to improve policies efficiently.
problem Learning from multiple suboptimal experts in reinforcement learning.
method Active policy improvement from multiple black-box oracles.
result MAPS and MAPS-SE achieve sample efficiency and state-wise imitation learning.
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
Develops a method to plan exploration that learns strong policies with fewer samples.
problem Lack of efficient exploration in reinforcement learning for real-world tasks.
method Plans an action sequence that maximizes information gain about the optimal trajectory.
result 2x fewer samples than exploration baselines and 200x fewer than model-free methods.
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.