A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New approach for open ad hoc teamwork using graph-based policy learning.
New method QMLE performs well in complex action spaces without policy gradients.
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
It is well known that quantifying uncertainty in the action-value estimates is crucial for efficient exploration in reinforcement learning. Ensemble sampling offers a relatively computationally tractable way of doing this using randomized value functions. However, it still requires a huge amount of computational resour…
MAGE optimizes policies using action gradients from model-based learning.
A new RL algorithm POWR learns world models to estimate action-values.
We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
A new method learns action representations for reinforcement learning.
Q()-Learning improves Q-Learning by separating action-value functions into different time scales.
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifte…
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicin…
Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gradient based deep RL algorithms efficiently. To effectively operate in multidimensional discrete acti…
New algorithms speed up inverse reinforcement learning by solving MDPs once.
Improves RL algorithms with two techniques.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
Imitation by observation is an approach for learning from expert demonstrations that lack action information, such as videos. Recent approaches to this problem can be placed into two broad categories: training dynamics models that aim to predict the actions taken between states, and learning rewards or features for com…
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
Develops a dynamic mean field theory for reinforcement learning.
Bayesian framework for learning optimal action-value function in MDPs.
QMIX combines per-agent values to create decentralised policies.
Transformers learn to predict chess moves with surprising accuracy and strength.
A new RL paradigm reduces state-action-value function approximation inefficiency.
Diffusion-QL uses diffusion models to improve offline RL performance.
QTRAN++ improves MARL performance in complex environments.
Latent variable models improve RL by facilitating efficient learning and exploration.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones f…
This work improves deep reinforcement learning robustness to adversarial state uncertainty.
Band-limited SAC improves learning efficiency and stability in simulated environments.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
Sparse representations have been shown to be useful in deep reinforcement learning for mitigating catastrophic interference and improving the performance of agents in terms of cumulative reward. Previous results were based on a two step process were the representation was learned offline and the action-value function w…
Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on …
A number of optimal decision problems with uncertainty can be formulated into a stochastic optimal control framework. The Least-Squares Monte Carlo (LSMC) algorithm is a popular numerical method to approach solutions of such stochastic control problems as analytical solutions are not tractable in general. This paper ge…
We propose a flexible method for estimating value functions in reinforcement learning without parametric assumptions.
New framework for modular reinforcement learning reduces sample complexity.
Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute t…
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
HyperAgent improves RL exploration in large-scale problems.
New RL method handles large state-action spaces with complex models.
Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in com…
An action selector associates, in a suitable way, to each compactly supported Hamiltonian on a symplectic manifold an action value of the Hamiltonian. Action selectors are known to exist for a broad class of symplectic manifolds. We show how the existence of an action selector leads to sharp energy capacity inequalitie…
Novel algorithm for Markov decision processes using rank-one approximation.
RLMM extends psychometric models to larger tasks.