A new multi-objective RL framework improves intrinsic exploration performance.
problem Sub-optimal exploration performance due to ad-hoc handling of intrinsic exploration.
method A multi-objective RL framework where both exploration and exploitation are optimized as separate objectives.
result EMU-Q method outperforms classic and other intrinsic RL methods on benchmarks.
Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which res…
The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.
problem Exploration in policy-gradient algorithms and its impact on policy optimization.
method Numerical optimization criteria and stochastic gradient analysis.
result Exploration techniques improve policy optimization by smoothing the learning objective and modifying gradient estimates.
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic r…
Efficient exploration for automatic subgoal discovery is a challenging problem in Hierarchical Reinforcement Learning (HRL). In this paper, we show that intrinsic motivation learning increases the efficiency of exploration, leading to successful subgoal discovery. We introduce a model-free subgoal discovery method base…
Here we explore a variety of properties of intrinsic flat convergence. We introduce the sliced filling volume and interval sliced filling volume and explore the relationship between these notions, the tetrahedral property and the disappearance of points under intrinsic flat convergence. We prove two new Gromov-Hausdorf…
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method Intrinsic rewards proportional to the entropy of future state-action features.
result The new objective leads to improved visitation of features within individual trajectories.
Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning. In the single-agent setting, this challenge is addressed by introducing intrinsic rewards that motivate agents to explore unseen regions of their state spaces; however, applying these techniques naively to the multi-age…
We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncert…
Novelty search in low-dimensional space improves sample efficiency in exploration tasks.
problem Efficient exploration in complex environments with sparse rewards.
method Combines model-based and model-free objectives to learn a low-dimensional representation. Uses intrinsic novelty rewards based on nearest neighbor distances in this space.
result Our approach achieves more sample-efficient exploration compared to strong baselines on various tasks.
Intrinsically motivated reinforcement learning aims to address the exploration challenge for sparse-reward tasks. However, the study of exploration methods in transition-dependent multi-agent settings is largely absent from the literature. We aim to take a step towards solving this problem. We present two exploration m…
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for …
New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
A method to train multi-agent reinforcement learning models without intrinsic rewards.
problem Training multi-agent reinforcement learning models with sparse rewards is challenging.
method A learning-based exploration strategy using variational graph autoencoder to generate initial states.
result The method improves the training and performance of multi-agent reinforcement learning models.
The paper characterizes biconservative surfaces in hyperbolic 4-space.
problem Characterizing biconservative surfaces in hyperbolic 4-space.
method Establishing intrinsic conditions and analyzing geometric properties.
result Biconservative surfaces in hyperbolic 4-space satisfy a specific intrinsic condition.
New RL approach uses future state and action visitation measures for better exploration.
problem Improving exploration in reinforcement learning.
method Intrinsic reward based on future state and action visitation measures, using contraction operators.
result Policies achieve good state-action space coverage and high performance.
SEMI uses multisensory incongruity to self-supervise exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning with sparse or missing rewards.
method SEMI incentivizes exploration by maximizing multisensory incongruity, measured in perception and action incongruity.
result SEMI improves sample efficiency and learns skills without external rewards.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
Study intrinsic volume forms on complex hypersurfaces.
problem Computing volume functionals on pseudoconvex hypersurfaces.
method Compute first and second variation formulae, explore infinite dimensional aspects.
result Discuss possible analogues of the affine isoperimetric inequality.
Graphs and their complements are intrinsically knotted.
problem Characterizing maximal linklessly embeddable graphs and their complements.
method Analyzing maximal linklessly embeddable graphs, deriving connected domination numbers, and proving intrinsic knotting properties.
result Complements of maximal linklessly embeddable graphs of order 12 and 15 are intrinsically knotted.
Study shows how to motivate AI agents like humans through learning from demonstrations.
problem Transfer human-like intrinsic motivation to AI agents in RL.
method Inverse RL approach to learn an exploration bonus from demonstrations.
result AI agents can learn and use complex exploration behaviors from demonstrations.
New L1-Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces L1-Coverage objective to enable efficient exploration and planning. result First computationally efficient algorithms for online reinforcement learning with low coverability.
In distributional reinforcement learning (RL), the estimated distribution of value function models both the parametric and intrinsic uncertainties. We propose a novel and efficient exploration method for deep RL that has two components. The first is a decaying schedule to suppress the intrinsic uncertainty. The second …
Autonomous reinforcement learning agents, like children, do not have access to predefined goals and reward functions. They must discover potential goals, learn their own reward functions and engage in their own learning trajectory. Children, however, benefit from exposure to language, helping to organize and mediate th…
This paper presents the Homeo-Heterostatic Value Gradients (HHVG) algorithm as a formal account on the constructive interplay between boredom and curiosity which gives rise to effective exploration and superior forward model learning. We envisaged actions as instrumental in agent's own epistemic disclosure. This motiva…
Robots learn actions and language through curiosity-driven self-exploration.
problem Efficient development of actions and language in infants and robots.
method Curiosity-driven self-exploration using Q-learning to amortize active inference.
result Curiosity-driven exploration enables faster learning and compositional generalization.
Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variation…
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as …
Capsule Networks improve autonomous navigation in sparse environments.
problem Challenges in reinforcement learning for sparse reward environments.
method Caps-EM pairs CapsNets with Advantage Actor Critic, using fewer parameters.
result Caps-EM achieves significant time improvements and fewer parameters compared to competing methods.
The paper proves stability of the positive mass theorem using intrinsic flat convergence.
problem Stability of the positive mass theorem in mathematical relativity.
method Intrinsic flat convergence of points and applications to stability.
result Revisits and strengthens the stability results for graphical hypersurfaces of Euclidean space.
We explore the intrinsic geometry of tangent bundles and properties of the mirror map.
problem Understanding the intrinsic geometry of tangent bundles and properties of the mirror map.
method Review and solve cohomological questions related to the mirror map and related operators.
result Solved some cohomological questions and raised other induced by the d_B operator.
Study local exploration on dynamic graphs with time-varying edges.
problem Learning optimal actions in a network with changing connections.
method Local explore-then-commit algorithms under a structural condition ensuring intrinsic walk stability.
result Sublinear expected regret for reward-aware strategies.
BYOL-Explore learns to explore visually-rich environments by predicting world dynamics.
problem Exploration in visually complex environments.
method Optimizes a single prediction loss in latent space to learn world representation, dynamics, and exploration policy.
result Achieves superhuman performance on Atari games with simpler design.
Investigates projections onto explicit subspaces and their variance effects.
problem Understanding the variance preservation in explicit subspace projections.
method Investigates projections onto explicit subspaces of varying dimensionality and analyzes the variance effects.
result Developed new bounds for Euclidean distances and inner products.
Interprets how intrinsic motivation shapes behavior in RL agents.
problem Understanding how intrinsic motivation influences behavior in reinforcement learning agents.
method Analyzed five RL agents in procedurally generated environments using various interpretability techniques.
result Curiosity-driven agents exhibit broader and more dynamic attention than extrinsically motivated agents.
Reinforcement learning algorithms struggle when the reward signal is very sparse. In these cases, naive random exploration methods essentially rely on a random walk to stumble onto a rewarding state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or …
Our work is a simple extension of the paper "Exploration by Random Network Distillation". More in detail, we show how to efficiently combine Intrinsic Rewards with Experience Replay in order to achieve more efficient and robust exploration (with respect to PPO/RND) and consequently better results in terms of agent perf…
The article presents a new entropy model for assessing stock market interest.
problem Assessing investor interest and market sentiment in exchange-traded securities.
method Intrinsic entropy model using actual trading data, without exogenous factors.
result Empirical evidence supports the model's ability to predict trading activity.
Study of potential Carroll structures and special Carrollian manifolds for null hypersurfaces.
problem Understanding intrinsic geometry of null hypersurfaces.
method Initiate study of potential Carroll structures and explore their relationship to special Carrollian manifolds.
result Initiate the study of potential Carroll structures and their relationship to special Carrollian manifolds.
A new reward learning module improves imitation learning in high-dimensional environments.
problem Challenges in high-dimensional environments for imitation learning.
method Generative model to generate intrinsic reward signals.
result Our method outperforms state-of-the-art IRL methods on Atari games.
The paper examines Wiener process for LID estimation methods.
problem Estimating local intrinsic dimension in high-dimensional datasets.
method Investigates recent LID estimation methods from a Wiener process perspective.
result Explains how methods behave under non-ideal conditions.
In this paper, we focus on a prediction-based novelty estimation strategy upon the deep reinforcement learning (DRL) framework, and present a flow-based intrinsic curiosity module (FICM) to exploit the prediction errors from optical flow estimation as exploration bonuses. We propose the concept of leveraging motion fea…
ADAC uses analogous policies to improve RL exploration without sacrificing stability.
problem Improving RL exploration without compromising stability and expressiveness.
method Disentangled actor-critic approach with analogous pairs of actors and critics.
result Empirical evaluation shows ADAC outperforms alternatives in challenging exploration tasks.
Universal AI seeks high-optionality states through empowerment and curiosity.
problem Understanding and optimizing AI behavior in uncertain environments.
method Unified framework combining AIXI and variational empowerment, showing how universal AI agents balance goal-directed behavior with uncertainty reduction curiosity.
result Self-AIXI asymptotically converges to AIXI performance and exhibits power-seeking behavior due to intrinsic motivations.
R3L uses planning algorithms to efficiently explore sparse reward environments.
problem Balancing exploration and exploitation in sparse reward reinforcement learning.
method Formulate exploration as a search problem using RRT, leverage demonstrations from initial solutions to refine RL policy.
result R3L outperforms classic and intrinsic exploration techniques, requiring fewer samples and achieving better asymptotic performance.
Study intrinsic motivation for synergistic tasks in reinforcement learning.
problem Sparse-reward synergistic tasks where multiple agents must work together.
method Propose incentivizing actions that affect the world in ways not achievable individually, using either true states or a dynamics model.
result Our approach yields more efficient learning than typical methods.
The paper studies new curvature properties in Finsler geometry.
problem Properties of projectively equivalent Finsler metrics and their curvature structures.
method Introducing new characterizations of quadratic curvature properties in Finsler manifolds.
result Novel insights into curvature behavior under generalized projective sprays.