A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Q()-Learning improves Q-Learning by separating action-value functions into different time scales.
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
New approach for open ad hoc teamwork using graph-based policy learning.
A new RL algorithm POWR learns world models to estimate action-values.
A new method learns action representations for reinforcement learning.
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifte…
New method QMLE performs well in complex action spaces without policy gradients.
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicin…
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gradient based deep RL algorithms efficiently. To effectively operate in multidimensional discrete acti…
MAGE optimizes policies using action gradients from model-based learning.
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…
Bayesian framework for learning optimal action-value function in MDPs.
QMIX combines per-agent values to create decentralised policies.
Transformers learn to predict chess moves with surprising accuracy and strength.
A new RL paradigm reduces state-action-value function approximation inefficiency.
Diffusion-QL uses diffusion models to improve offline RL performance.
It is well known that quantifying uncertainty in the action-value estimates is crucial for efficient exploration in reinforcement learning. Ensemble sampling offers a relatively computationally tractable way of doing this using randomized value functions. However, it still requires a huge amount of computational resour…
QTRAN++ improves MARL performance in complex environments.
New algorithms speed up inverse reinforcement learning by solving MDPs once.
Latent variable models improve RL by facilitating efficient learning and exploration.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones f…
We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…
Band-limited SAC improves learning efficiency and stability in simulated environments.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
Sparse representations have been shown to be useful in deep reinforcement learning for mitigating catastrophic interference and improving the performance of agents in terms of cumulative reward. Previous results were based on a two step process were the representation was learned offline and the action-value function w…
Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on …
We propose a flexible method for estimating value functions in reinforcement learning without parametric assumptions.
New framework for modular reinforcement learning reduces sample complexity.
Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute t…
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
New RL method handles large state-action spaces with complex models.
Develops a dynamic mean field theory for reinforcement learning.
An action selector associates, in a suitable way, to each compactly supported Hamiltonian on a symplectic manifold an action value of the Hamiltonian. Action selectors are known to exist for a broad class of symplectic manifolds. We show how the existence of an action selector leads to sharp energy capacity inequalitie…
RLMM extends psychometric models to larger tasks.
Study shows convex contact spheres resemble contact ellipsoids.
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
Study improves understanding of why agentic theorem provers succeed.
We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive -functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …
UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.
We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…