arXiv research

A new RL paradigm reduces state-action-value function approximation inefficiency.

New RL method handles large state-action spaces with complex models.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

Novel framework proves fast RL convergence in continuous spaces.

QR-MIX models joint state-action values as a distribution to handle randomness in MARL.

New method improves deep policy gradient algorithms by learning relative state values.

Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.

Develops a dynamic mean field theory for reinforcement learning.

UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structu…

A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive $q$-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …

The paper tackles reward-relevance in offline RL with sparse decision dynamics.

New RL approach uses future state and action visitation measures for better exploration.

Latent variable models improve RL by facilitating efficient learning and exploration.

Proposes a method to enhance exploration in RL using temporal difference uncertainties.

New RL method reduces sample complexity for large state-action spaces.

This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.

PBVFs generalize across policies using learned value functions.

New algorithm for online learning in episodic MDPs with convex objectives.

Band-limited SAC improves learning efficiency and stability in simulated environments.

Paper improves RL algorithms with known optimal Q-value predictions.

We introduce a novel Deep Reinforcement Learning (DRL) algorithm called Deep Quality-Value (DQV) Learning. DQV uses temporal-difference learning to train a Value neural network and uses this network for training a second Quality-value network that learns to estimate state-action values. We first test DQV's update rules…

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…

GUM tackles MARL by avoiding overestimation through state-marginal restriction.

Study risk-sensitive reinforcement learning with optimized certainty equivalents.

New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.

The paper proposes a principle for dynamically adjusting the granularity of reinforcement learning abstractions.

Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some glo…

Deep neural networks can estimate Q-values efficiently on low-dimensional state-action spaces.

Breaks the hardness conjecture for batch RL with a novel tournament-based approach.

We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…

We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning. Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value f…

ESAC improves reinforcement learning by lookahead and intuition.

We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions. Our contributions include: (1) A new estimator, MWL, that directly estimates importance ratios over the state-action distributions, removing …

Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.

Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.

Develops a new reinforcement learning framework for complex control problems.

Hierarchical Modular Reinforcement Learning (HMRL), consists of 2 layered learning where Profit Sharing works to plan a prey position in the higher layer and Q-learning method trains the state-actions to the target in the lower layer. In this paper, we expanded HMRL to multi-target problem to take the distance between …

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…

MO2 learns useful behaviours from past experience for new tasks.

We seek to align agent policy with human expert behavior in a reinforcement learning (RL) setting, without any prior knowledge about dynamics, reward function, and unsafe states. There is a human expert knowing the rewards and unsafe states based on his preference and objective, but querying that human expert is expens…

We establish that an optimistic variant of Q-learning applied to a fixed-horizon episodic Markov decision process with an aggregated state representation incurs regret $\tilde{\mathcal{O}}(\sqrt{H^5 M K} + εHK)$, where $H$ is the horizon, $M$ is the number of aggregate states, $K$ is the number of episodes, and $ε$ is …

Study optimal offline RL with uncertainty sets and distribution shifts.

This paper makes one step forward towards characterizing a new family of \textit{model-free} Deep Reinforcement Learning (DRL) algorithms. The aim of these algorithms is to jointly learn an approximation of the state-value function ($V$), alongside an approximation of the state-action value function ($Q$). Our analysis…

Network slicing promises to provision diversified services with distinct requirements in one infrastructure. Deep reinforcement learning (e.g., deep $\mathcal{Q}$-learning, DQL) is assumed to be an appropriate algorithm to solve the demand-aware inter-slice resource management issue in network slicing by regarding the …

Proposes a value-based method for continuous control without an actor.

A new method estimates multi-dimensional value distributions using Hilbert space embeddings.