This paper improves MADDPG's performance in discrete grid-world scenarios.
problem MADDPG struggles with discrete action spaces in grid-world tasks.
method Integrates alternative Gumbel-Softmax estimators into MADDPG.
result One alternative estimator significantly improves performance in grid-world tasks.
Graph-based state representation improves deep RL performance.
problem High sample-complexity and starting with a good input representation in deep RL.
method Exploiting the graph structure of MDPs for effective state representation learning.
result Graph-based node representation methods outperform matrix-based methods in grid-world navigation tasks.
APES simplifies reinforcement learning environment design in Python.
problem Creating and simulating reinforcement learning environments.
method Introduces APES, a Python toolbox for 2D grid-world environments.
result Equips reinforcement learning agents with customizable field of vision and item/reward interactions.
Paper tackles reinforcement learning generalization through invariant policy optimization.
problem Learning policies that generalize beyond training domains.
method Invariant policy optimization principle and novel learning algorithm IPO.
result Significant improvements in generalization performance on unseen domains.
The paper introduces MDP homomorphic networks for faster reinforcement learning.
problem Current reinforcement learning approaches do not exploit symmetries in the joint state-action space.
method Equivariant neural networks with group-structured symmetries (reflections, rotations).
result MDP homomorphic networks converge faster than unstructured baselines on various tasks.
We designed a grid world task to study human planning and re-planning behavior in an unknown stochastic environment. In our grid world, participants were asked to travel from a random starting point to a random goal position while maximizing their reward. Because they were not familiar with the environment, they needed…
Resolves spurious correlations in causal models via intervention design.
problem Spurious correlations lead to incorrect causal models in reinforcement learning environments.
method Proposes a method to design interventions that improve causal models by incentivizing agents to find errors.
result Experimental results show improved causal models compared to baselines.
Successor Options discovers reusable skills using landmark states.
problem Discovering reusable skills in reinforcement learning.
method Leverages Successor Representations to build a state space model and learns intra-option policies using a novel pseudo-reward.
result Demonstrates the approach's efficacy on grid-worlds and high-dimensional robotic control environments.
Paper proposes PRR network for better experience reuse in reinforcement learning.
problem Efficient experience reuse in reinforcement learning across multiple granularities.
method Proposes PRR network trained on multi-level architecture to extract and store experience.
result PRR network leads to better experience reuse and improved performance.
Curiosity-Critic improves world model training by focusing on cumulative prediction error.
problem Training world models with intrinsic rewards that consider cumulative prediction error.
method Curiosity-Critic uses a surrogate reward based on the difference between current and asymptotic prediction errors, estimated online by a co-trained critic.
result Curiosity-Critic outperforms other methods in training speed and final world model accuracy.
New method identifies critical states to improve RL agent explainability and speed.
problem Challenges in RL agent explainability and action selection timing.
method Identify critical states based on action-based variance in Q-function, prioritize exploitation on these states.
result Identified critical states accelerate RL in grid worlds and deep RL tasks.
Framework discovers sub-goals for better exploration in RL tasks.
problem Improving exploration in sequential decision making under partial observability.
method Variational intrinsic control framework with information theoretic regularizer.
result Sub-goals discovered without explicit supervision lead to better exploration and sample efficiency.
Accelerates Q-learning by updating targets more efficiently.
problem Improving convergence speed in Q-learning algorithms.
method Proposes an accelerated target update scheme based on momentum methods.
result The proposed algorithms converge faster than vanilla Q-learning.
We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…
According to Dennett, the same system may be described using a `physical' (mechanical) explanatory stance, or using an `intentional' (belief- and goal-based) explanatory stance. Humans tend to find the physical stance more helpful for certain systems, such as planets orbiting a star, and the intentional stance for othe…
Graph neural network executes value iteration for flexible environments.
problem Value iteration in flexible environments with direct supervision.
method Graph Neural Network (GNN) executing value iteration algorithm.
result GNN accurately models value iteration across diverse environments.
A new IRL model recovers reward and state structure from expert demonstrations.
problem Limitation of classical maximum entropy model in capturing state structure.
method Generalized maximum causal entropy for IRL models.
result Empirically outperforms classical models in recovering reward and state structure.
Improved exploration in SAC using Normalizing Flows policies.
problem Brittleness and inefficiency of DRL algorithms in continuous action spaces.
method Introducing Normalizing Flow policies within the SAC framework to learn more expressive policies.
result Increased stability and better exploration in sparse reward settings.
A method for Bayes-Adaptive Deep RL using meta-learning.
problem Maximizing expected return in unknown environments with uncertainty.
method variBAD: meta-learning for approximate inference and task uncertainty.
result variBAD achieves higher online return than existing methods in MuJoCo domains.
Curiosity-driven agent predicts future state values to learn faster.
problem Sparse reward learning in reinforcement learning.
method The agent predicts future state values and uses consistency with current values as a regularization term.
result The extended agent learns significantly faster in sparse reward settings.
In this semi-tutorial paper, we first review the information-theoretic approach to account for the computational costs incurred during the search for optimal actions in a sequential decision-making problem. The traditional (MDP) framework ignores computational limitations while searching for optimal policies, essential…
Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor ma…
Backplay improves RL efficiency with demonstrations.
problem Improving sample efficiency in sparse reward environments.
method Constructs a curriculum using a single demonstration to start training near the end of the demo.
result Backplay outperforms other methods in grid worlds and a complex game.
Paper proposes methods to handle missing data in online RL, improving efficiency and uncertainty capture.
problem Missing data in online RL poses challenges due to the need to impute and act at each time step.
method Proposes fully online imputation ensembles and multiple imputation pathways to balance uncertainty and efficiency.
result Preliminary evidence suggests multiple imputation pathways can be a useful framework for simple and efficient online missing data RL methods.
The paper proposes a method to infer Q-values online with Q-Learning.
problem High variance and instability in reinforcement learning algorithms.
method Adapting FCLT for a modified Q-learning approach and constructing confidence intervals.
result The proposed method provides more stable and reliable inference of Q-values.
The paper uses potential functions to help reinforcement learning agents learn optimal policies.
problem Learning optimal stochastic policies in reinforcement learning.
method Augmenting the reward with potential functions and applying these to policy gradient algorithms.
result Potential-based reward shaping preserves optimality of stochastic policies and speeds up learning.
Meta-learner infers subtask graph to adapt quickly to unknown tasks.
problem Adapting to unknown tasks with unknown subtask dependencies in few-shot RL.
method Meta-learner with Subtask Graph Inference (MSGI) and UCB-inspired intrinsic reward.
result Accurately infers latent task parameters and adapts more efficiently.
Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
APT-Gen generates tasks to help RL learn in hard problems.
problem Learning in hard exploration problems.
method APT-Gen uses a task generator to create tasks from a parameterized space, balancing performance and similarity to target tasks.
result APT-Gen outperforms baselines in grid world and robotic manipulation tasks.
Deep neural network learns discrete state abstractions for efficient planning.
problem Efficient sequential decision making in large state spaces.
method Information bottleneck method for learning approximate bisimulations using deep neural encoders and action-conditioned HMM.
result Trained method efficiently plans for unseen goals in multi-goal reinforcement learning.
ICEE learns new RL tasks in less time with a Transformer model.
problem Efficient in-context policy learning for reinforcement learning.
method In-context Exploration-Exploitation (ICEE) algorithm that optimizes efficiency without explicit Bayesian inference.
result ICEE solves Bayesian optimization problems as efficiently as Gaussian process biased methods but in significantly less time.
Option Encoder compresses reinforcement learning options into a policy basis.
problem Redundant options in reinforcement learning frameworks.
method Auto-encoder framework with constrained weights to discover a policy basis.
result Option Encoder reduces the number of options while maintaining performance.
Projective simulation (PS) is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speed-up in deliberat…
Paper proposes a deep RL algorithm that can learn from forbidden actions.
problem Learning from forbidden actions in constrained environments.
method Enhanced DQN with safety loss for learning from unsafe actions.
result Reduces hit constraints and accelerates convergence in experiments.
Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision making under uncertainty. The classical approaches for solving MDPs are well known and have been widely studied, some of which rely on approximation techniques to solve MDPs with large state space and/or action space. However…
A new algorithm finds optimal solutions for constrained decision processes.
problem Optimizing state-value functions with constraints in CMDPs.
method Gradient-Aware Search (GAS) exploiting PWLC structure.
result GAS converges faster and more reliably than existing methods.
Here we propose using the successor representation (SR) to accelerate learning in a constructive knowledge system based on general value functions (GVFs). In real-world settings like robotics for unstructured and dynamic environments, it is infeasible to model all meaningful aspects of a system and its environment by h…
Improves RL planning by proposing sub-goals hierarchically.
problem Sequential planning assumption in RL.
method Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS).
result Improves navigation and control tasks.
HRL4IN tackles interactive navigation tasks with mobile manipulators, improving efficiency and performance.
problem Interactive Navigation tasks require mobile manipulators to perform various actions, but choosing the right part of the embodiment is inefficient.
method HRL4IN uses a hierarchical reinforcement learning architecture to handle heterogeneous phases of navigation and manipulation, selecting the appropriate part of the embodiment for each phase.
result HRL4IN significantly outperforms flat PPO and HAC in terms of task performance and energy efficiency.
New mirror maps improve PMD performance in reinforcement learning.
problem Limited exploration of PMD's full potential due to focus on negative entropy.
method Evolutionary strategies to identify and learn more efficient mirror maps.
result Learned mirror maps outperform negative entropy in various environments.
New algorithm improves self-play reinforcement learning for competitive games.
problem Inefficient opponent selection in self-play reinforcement learning.
method Intelligently selects opponents based on adversarial rules derived from saddle point optimization.
result Algorithm converges to approximate equilibrium with high probability in convex-concave games.
Jelly Bean World tests never-ending learning in complex environments.
problem Current machine learning systems are specialized and limited in adaptability.
method Developed Jelly Bean World testbed for complex, non-stationary environments.
result Demonstrates the need for more adaptable machine learning systems.
CF-GPS learns policies from logged data by considering counterfactual outcomes.
problem Learning policies from limited real experience in complex environments.
method Assumes logged real experience and models counterfactual outcomes. Uses structural causal models for evaluation.
result Improves policy evaluation and search results on a grid-world task.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
Deep RL policies can leak private information from trained policies.
problem Privacy leakage in deep reinforcement learning models.
method Environment dynamics search via genetic algorithm and candidate inference based on shadow policies.
result 95.83% average recovery rate of floor plans from trained Grid World navigation DRL agents.
New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
SMAC benchmarks StarCraft II multi-agent micromanagement challenges.
problem Benchmarking cooperative multi-agent reinforcement learning.
method Proposes StarCraft Multi-Agent Challenge (SMAC) for evaluation.
result SMAC provides a standard benchmark for multi-agent RL.
Expanding models in neural fictitious play improves reinforcement learning efficiency and robustness.
problem Forgetting old opponents after training new ones in reinforcement learning.
method Train a single model with sub-models and a selector, expanding the model with new sub-models and updating the selector to maintain a behavior strategy.
result Improves learning efficiency and robustness of neural fictitious play.