Enhanced reinforcement learning using ensemble methods.
problem Improving reinforcement learning performance.
method Distributional reinforcement learning with ensemble group-aided training.
result Ensemble methods lead to more robust and efficient learning.
Study OOD generalization in meta-reinforcement learning using information theory.
problem Understanding how meta-reinforcement learning handles distribution shifts.
method Information-theoretic analysis of Markov Decision Processes and gradient-based algorithms.
result Established fine-grained generalization bounds for meta-reinforcement learning.
Paper introduces a new distributional successor measure for reinforcement learning.
problem Learning the distributional consequences of behavior in reinforcement learning.
method Formulates distributional successor measure as a distribution over distributions, proposes algorithm to learn it from data.
result Demonstrates zero-shot risk-sensitive policy evaluation.
Paper robustifies reinforcement learning with risk-averse methods.
problem Making predictions robust to changes in system dynamics or rewards.
method Approximates Robust Reinforcement Learning using Φ-divergence and Risk-Averse formulation. result Classical Reinforcement Learning can be robustified using standard deviation penalization.
Bayesian meta-reinforcement learning improves over point estimates with Laplace approximation.
problem Improving meta-reinforcement learning by providing full posterior distributions.
method Augmenting point estimates with Laplace approximation for full posterior distributions.
result Our method performs similarly to variational baselines with fewer parameters.
The paper proves the convergence of Q-value for Gaussian rewards.
problem Existing proofs cannot guarantee convergence of the Q-function for Gaussian rewards.
method Using the central limit theorem and relaxing the condition to E[r(s,a)2]<∞. result Proves the convergence of the Q-function under the condition of E[r(s,a)2]<∞. Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributiona…
We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies multiple previous methods in exploration. We also derive a practical algorithm that achieves efficien…
In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average …
The paper explores solutions to the distributional Bellman equation in reinforcement learning.
problem Distributional reinforcement learning considers complete return distributions, not just expected returns.
method Study existence and uniqueness of solutions to general distributional Bellman equations, linking them to multivariate affine equations.
result Any solution to a distributional Bellman equation can be derived from a multivariate affine distributional equation.
This work analyzes how multi-agent reinforcement learning can bridge the gap to reality in distributed multi-robot systems.
problem Collaborative learning in distributed multi-robot systems with varying sensors and actuators.
method Simulation-based analysis using PPO and Bullet physics engine, considering different types of perturbations.
result PPO's robustness is affected by the presence of different types of perturbations and the number of agents experiencing them.
This paper explains why distributional reinforcement learning is better than vanilla RL using small-loss bounds.
problem Understanding when and why distributional reinforcement learning (DistRL) is superior to vanilla reinforcement learning (RL).
method The paper uses small-loss bounds to explain the benefits of DistRL, proposing algorithms and proving bounds for different RL settings.
result Distributional reinforcement learning (DistRL) outperforms vanilla RL when optimal costs are small, as shown by small-loss bounds.
New method uses DistRL to estimate entire payoff distribution for financial derivatives.
problem Traditional methods focus on expected option value; this tackles risk-aware pricing.
method Reinterprets and proposes a framework using Distributional Reinforcement Learning (DistRL).
result Demonstrates enhanced risk-aware pricing and uncertainty quantification on Asian options.
New algorithm for RL using mean embeddings of return distributions.
problem Improving reinforcement learning algorithms for dynamic programming.
method Mean embeddings of return distributions, novel algorithms for RL.
result Asymptotic convergence and improved performance in deep RL.
New RL algorithm minimizes distributional learning error.
problem Improving distributional reinforcement learning for better error minimization.
method Proposes a new model-based algorithm with theoretical minimax optimality.
result Proves minimax optimality for approximating return distributions.
In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established …
We analyze reinforcement learning algorithms using a distributional approach.
problem Theoretical analysis of reinforcement learning algorithms for constant step-sizes.
method Distributional approach to theoretical analyses of reinforcement learning algorithms.
result TD(λ) and Q-Learning have contractive update rules in the space of distributions of functions, leading to exponentially fast convergence. Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.
problem Improving reinforcement learning by capturing environmental stochasticity and addressing infinite dimensionality.
method Introduces Bellman unbiasedness and proposes SF-LSVI algorithm for provably efficient distributional reinforcement learning.
result Achieves a tight regret bound of O(d_E H^3/2 √K) for distributional reinforcement learning.
This work simplifies choosing reinforcement-learning algorithms.
problem Challenging selection of reinforcement-learning algorithms.
method Structured overview and guidelines for choosing methods.
result Provides structured guidelines for algorithm selection.
Reinforcement learning (RL) tasks are challenging to implement, execute and test due to algorithmic instability, hyper-parameter sensitivity, and heterogeneous distributed communication patterns. We argue for the separation of logical component composition, backend graph definition, and distributed execution. To this e…
New algorithms handle heavy-tailed rewards in reinforcement learning.
problem Learning from heavy-tailed rewards in reinforcement learning.
method Robust mean estimation techniques for constructing algorithms.
result Near-optimal regret bounds achieved in heavy-tailed reward settings.
Algorithm learns new tasks efficiently from past experience.
problem Lack of robustness to distributional shift in meta-reinforcement learning.
method Model Identification and Experience Relabeling (MIER) using dynamics models.
result Efficient extrapolation to out-of-distribution tasks.
Improves reinforcement learning policies for robustness.
problem Lack of robustness in reinforcement learning policies.
method Risk-aware Distributional Reinforcement Learning (SDPG) with CVaR.
result Risk-averse policies achieve robustness against disturbances.
New method extends fitted Q-evaluation for distributional off-policy reinforcement learning.
problem Estimating return distribution in reinforcement learning using offline data.
method Developed a set of guiding principles and new FDE methods with theoretical justification.
result FDE methods outperform existing approaches in simulations and real-world games.
New method handles large reward variations in reinforcement learning.
problem Optimal policy not achievable with existing methods for non-deterministic processes.
method Introduces conjugated distributional operator for handling real returns.
result Guaranteed theoretical convergence for a wide class of transformations.
An ensemble method enhances cryptocurrency trading strategies using deep reinforcement learning.
problem Improving generalization performance in stochastic cryptocurrency trading environments.
method Model selection and mixture distribution policy to ensemble deep reinforcement learning models.
result Improved out-of-sample performance compared to benchmarks.
Diverse projection ensembles improve distributional reinforcement learning.
problem Learning the distribution of returns in reinforcement learning.
method Combining multiple projection methods to improve model diversity and exploration.
result Diverse projection ensembles lead to significant performance improvements in exploration tasks.
Generalization and adaptation of learned skills to novel situations is a core requirement for intelligent autonomous robots. Although contextual reinforcement learning provides a principled framework for learning and generalization of behaviors across related tasks, it generally relies on uninformed sampling of environ…
Paper establishes DRL for high-dimensional rewards.
problem Intractable reinforcement learning with high-dimensional rewards.
method Theoretical foundations and a novel DRL algorithm.
result Bellman operator contraction in high-dimensional spaces.
We consider the problem of detecting out-of-distribution (OOD) samples in deep reinforcement learning. In a value based reinforcement learning setting, we propose to use uncertainty estimation techniques directly on the agent's value estimating neural network to detect OOD samples. The focus of our work lies in analyzi…
Training intelligent agents through reinforcement learning is a notoriously unstable procedure. Massive parallelization on GPUs and distributed systems has been exploited to generate a large amount of training experiences and consequently reduce instabilities, but the success of training remains strongly influenced by …
Quantile TD learning outperforms classical TD learning for value estimation.
problem Temporal-difference learning in reinforcement learning.
method Quantile Temporal-Difference Learning (QTD) for policy evaluation.
result QTD offers superior performance to classical TD learning, even in tabular settings.
Simpler one-step distributional RL framework for control.
problem Lack of a unified theory for DistrRL in control.
method One-step distributional reinforcement learning (OS-DistrRL) framework.
result Unified theory for policy evaluation and control.
New algorithms improve distributional TD learning with linear approximations.
problem Estimating return distributions in reinforcement learning.
method Fine-grained analysis of linear-categorical Bellman equation, variance reduction techniques.
result Tight sample complexity bounds for distributional TD learning with linear approximations.
New algorithms for efficient return distribution approximation in reinforcement learning.
problem Efficiently approximating unknown return distributions in reinforcement learning.
method Introduced novel distributional dynamic programming algorithms for arbitrary probabilistic reward mechanisms.
result Proved error bounds for the algorithms in Wasserstein and Kolmogorov--Smirnov distances.
Paper uses reinforcement learning to optimize bid-ask spreads in OTC markets.
problem Optimizing bid-ask spreads in over-the-counter markets with dynamic order sizes.
method Reinforcement learning to solve high-dimensional stochastic control problem.
result Optimal bid-ask spreads follow a Gaussian distribution under certain conditions.
Paper develops robust policy evaluation for reinforcement learning with outlier and heavy-tailed rewards.
problem Outlier contamination and heavy-tailed rewards in reinforcement learning.
method Develops a fully online robust policy evaluation procedure and efficient statistical inference.
result Establishes the Bahadur-type representation of the estimator and develops an online inference procedure.
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
IDAC improves reinforcement learning efficiency by modeling implicit distributions.
problem Improving sample efficiency in reinforcement learning algorithms.
method IDAC uses two DGNs for a distributional critic and a semi-implicit actor to model implicit policy distributions.
result IDAC outperforms state-of-the-art algorithms on OpenAI Gym environments.
WAPPO optimizes feature distributions for better visual transfer in RL.
problem Improving visual transfer in reinforcement learning.
method WAPPO uses Wasserstein Confusion to minimize feature distribution distance.
result WAPPO outperforms previous methods in visual transfer across different environments.
Unified reinforcement learning and stochastic processes with action-driven processes.
problem Combining reinforcement learning and stochastic processes for efficient control.
method Action-driven processes, leveraging control-as-inference, and minimizing Kullback-Leibler divergence.
result Action-driven processes unify reinforcement learning and stochastic processes, equivalent to maximum entropy reinforcement learning.
Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depe…
DRL agents perform poorly at high decision frequencies, but a new algorithm improves performance.
problem DRL agents struggle at high decision frequencies, leading to poor performance.
method Proved that DRL agents' action-conditioned return distributions collapse to their policy's return distribution as decision frequency increases. Defined superiority as a probabilistic generalization of advantage for high-frequency value-based RL.
result Proper modeling of superiority distribution improves performance of controllers at high decision frequencies.
Optimizes trade execution with reinforcement learning for limit orders.
problem Maximizing revenue in a limit order book with market and limit orders.
method Formulated as a dynamic allocation task, uses multivariate logistic-normal distributions for efficient training.
result Outperforms traditional strategies in simulated environments.
We solve continuous-time reinforcement learning using distributional Hamilton-Jacobi-Bellman equations.
problem Predicting the distribution of returns in continuous-time, stochastic environments.
method We derive a distributional Hamilton-Jacobi-Bellman equation for Itô diffusions and Feller-Dynkin processes, and propose an algorithm based on a JKO scheme.
result We propose an online control algorithm that can be used to approximately solve the distributional HJB equation.
SPQR improves Q-ensemble diversity in reinforcement learning.
problem Overestimation bias in Q-learning for complex tasks.
method Introduces SPQR for Q-ensemble independence regularization.
result SPQR outperforms baseline algorithms in online and offline RL benchmarks.
Taylor expansions improve reinforcement learning policies.
problem Improving reinforcement learning policy optimization.
method Taylor expansion policy optimization.
result Taylor expansions enhance performance of distributed algorithms.
PEOC uses policy entropy to detect untrained states in RL.
problem Detecting untrained states in reinforcement learning for safety.
method Policy entropy based one-class classifier.
result PEOC is highly competitive and reliable.