arXiv research

A new RL paradigm reduces state-action-value function approximation inefficiency.

New RL method handles large state-action spaces with complex models.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

QR-MIX models joint state-action values as a distribution to handle randomness in MARL.

Novel framework proves fast RL convergence in continuous spaces.

Develops a dynamic mean field theory for reinforcement learning.

A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…

New method improves deep policy gradient algorithms by learning relative state values.

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…

Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.

The paper proposes a principle for dynamically adjusting the granularity of reinforcement learning abstractions.

UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…

MO2 learns useful behaviours from past experience for new tasks.

New RL approach uses future state and action visitation measures for better exploration.

Latent variable models improve RL by facilitating efficient learning and exploration.

A new method estimates multi-dimensional value distributions using Hilbert space embeddings.

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structu…

The paper tackles reward-relevance in offline RL with sparse decision dynamics.

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive $q$-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …

Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.

New method for natural policy gradients converges linearly.

Proposes a method to enhance exploration in RL using temporal difference uncertainties.

Band-limited SAC improves learning efficiency and stability in simulated environments.

New method recovers diverse policies from expert data using state-action pair weighting.

New RL method reduces sample complexity for large state-action spaces.

Adaptive discretization improves model-based RL in large spaces.

This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.

We introduce a novel Deep Reinforcement Learning (DRL) algorithm called Deep Quality-Value (DQV) Learning. DQV uses temporal-difference learning to train a Value neural network and uses this network for training a second Quality-value network that learns to estimate state-action values. We first test DQV's update rules…

GUM tackles MARL by avoiding overestimation through state-marginal restriction.

Paper improves RL algorithms with known optimal Q-value predictions.

Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute t…

New framework for RL transfer learning with state-action mismatch.

Develops a new reinforcement learning framework for complex control problems.

Study risk-sensitive reinforcement learning with optimized certainty equivalents.

Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.

Real-world applications require RL algorithms to act safely. During learning process, it is likely that the agent executes sub-optimal actions that may lead to unsafe/poor states of the system. Exploration is particularly brittle in high-dimensional state/action space due to increased number of low-performing actions. …

Study evaluates reinforcement learning for trading diverse stocks, finds Q-learning outperforms.

A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.

New algorithm for online learning in episodic MDPs with convex objectives.

New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.

Study optimal offline RL with uncertainty sets and distribution shifts.

ESAC improves reinforcement learning by lookahead and intuition.

Hierarchical Modular Reinforcement Learning (HMRL), consists of 2 layered learning where Profit Sharing works to plan a prey position in the higher layer and Q-learning method trains the state-actions to the target in the lower layer. In this paper, we expanded HMRL to multi-target problem to take the distance between …

We establish that an optimistic variant of Q-learning applied to a fixed-horizon episodic Markov decision process with an aggregated state representation incurs regret $\tilde{\mathcal{O}}(\sqrt{H^5 M K} + εHK)$, where $H$ is the horizon, $M$ is the number of aggregate states, $K$ is the number of episodes, and $ε$ is …

PBVFs generalize across policies using learned value functions.

The paper introduces a new intrinsic reward method for exploration in reinforcement learning.

We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning. Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value f…