Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “action-value methods”
New approach for open ad hoc teamwork using graph-based policy learning.
problem Designing autonomous agents to collaborate with changing teams without prior coordination.
method Graph-based policy learning to adapt to dynamic team compositions.
result Successfully models the effects of other agents, leading to robust adaptation and superior performance.
A new RL algorithm POWR learns world models to estimate action-values.
problem Inaccessibility of explicit action-value functions in RL.
method Learning a world model using conditional mean embeddings and deriving action-value function via matrix operations.
result POWR algorithm converges to global optimum with proven rates.
A new method learns action representations for reinforcement learning.
problem Efficient action-value estimation in reinforcement learning.
method Action hypergraph networks framework for learning action representations.
result Hypergraph Q-networks show effectiveness on various domains.
New method QMLE performs well in complex action spaces without policy gradients.
problem Why policy gradients outperform action-value methods in complex action spaces.
method QMLE framework for action-value methods based on three principles.
result QMLE performs comparably to policy gradient methods in complex action spaces.
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifte…
Q()-Learning improves Q-Learning by separating action-value functions into different time scales.
problem Q-Learning struggles with bias-variance trade-off, especially in long-term rewards.
method Introduces Q()-Learning, extending TD() to decompose Q()-function into distinct discount factors.
result Q()-Learning achieves better stability and scalability, especially for long-term tasks.
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
problem Overestimation and high variance in Q-Learning cause learning algorithms to diverge in complex environments.
method Deep Weighted Q-Learning (Deep WQL) uses Dropout and Monte Carlo sampling to approximate WQL's weights and reduce bias.
result Deep WQL reduces bias and improves performance on benchmarks compared to existing methods.
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
QMIX combines per-agent values to create decentralised policies.
problem Training decentralised policies from centralised learning.
method QMIX uses a mixing network to estimate joint action-values as a monotonic combination of per-agent values.
result QMIX significantly outperforms existing methods on the StarCraft Multi-Agent Challenge (SMAC).
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
Diffusion-QL uses diffusion models to improve offline RL performance.
problem Offline RL struggles with function approximation errors on out-of-distribution actions.
method Diffusion-QL represents the policy as a conditional diffusion model and optimizes action-values.
result Diffusion-QL achieves state-of-the-art performance on D4RL benchmark tasks.
GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicingcs.LG
Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicin…
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcs.LG
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones f…
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
problem Determining the minimum number of queries needed for sound planners in MDPs with linear function approximation.
method Analyzing fixed-horizon and discounted MDPs with a generative model, showing lower bounds on the number of queries required.
result Sound planners need at least exponential number of queries in both fixed-horizon and discounted settings.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
We propose a flexible method for estimating value functions in reinforcement learning without parametric assumptions.
problem Lack of interpretability in reinforcement learning models, especially in healthcare applications.
method Nonparametric additive model using local kernel regression and basis expansion.
result Personalized, adaptive recommendations for postoperative recovery.
New framework for modular reinforcement learning reduces sample complexity.
problem Achieving independent credit assignment in reinforcement learning.
method Defining modular credit assignment as minimizing algorithmic mutual information, introducing modularity criterion for causal analysis.
result Single-step temporal difference action-value methods meet the modularity criterion, improving sample efficiency.
Bayesian framework for learning optimal action-value function in MDPs.
problem Uncertainty quantification in MDPs for optimal decision-making strategies.
method Full Bayesian framework including modelling, inference, and decision-making.
result Demonstrates exploration benefits of posterior sampling in MDPs.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
Transformers learn to predict chess moves with surprising accuracy and strength.
problem Training transformers on chess to predict moves accurately.
method Large-scale chess dataset (10M games), supervised learning with up to 270M parameters.
result Transformers can predict action-values for novel boards with high accuracy.
It is well known that quantifying uncertainty in the action-value estimates is crucial for efficient exploration in reinforcement learning. Ensemble sampling offers a relatively computationally tractable way of doing this using randomized value functions. However, it still requires a huge amount of computational resour…
Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute t…
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
QTRAN++ improves MARL performance in complex environments.
problem Poor empirical performance of QTRAN in complex environments.
method Stabilizing training objective, removing role separation, and introducing a multi-head mixing network.
result QTRAN++ achieves state-of-the-art performance in the Starcraft Multi-Agent Challenge (SMAC).
New algorithms speed up inverse reinforcement learning by solving MDPs once.
problem Slow convergence in Maximum Entropy Inverse Reinforcement Learning.
method Deep Inverse Q-learning with constraints exploiting Q-learning.
result Up to several orders of magnitude speedup compared to existing methods.
New RL method handles large state-action spaces with complex models.
problem Complex models and large state-action spaces in reinforcement learning.
method π-KRVI, an optimistic modification of least-squares value iteration using kernel ridge regression.
result First order-optimal regret guarantees under general settings, improving over state of the art.
Latent variable models improve RL by facilitating efficient learning and exploration.
problem Improving sample efficiency in reinforcement learning.
method Representation view of latent variable models for state-action value functions, incorporating kernel embeddings and UCB exploration.
result Established sample complexity of the proposed approach in online and offline settings, demonstrated superior performance in benchmarks.
Mean Actor Criticstat.ML
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…
We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used…
Band-limited SAC improves learning efficiency and stability in simulated environments.
problem Improving sample efficiency and stability in SAC algorithms.
method Artificially bandlimiting the target critic's spatial resolution using a convolutional filter.
result Bandlimited SAC outperforms classic twin-critic SAC in various Gym environments and is more stable.
Sparse representations have been shown to be useful in deep reinforcement learning for mitigating catastrophic interference and improving the performance of agents in terms of cumulative reward. Previous results were based on a two step process were the representation was learned offline and the action-value function w…
Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on …
We consider tackling a single-agent RL problem by distributing it to learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local plan…
End-to-end deep reinforcement learning has enabled agents to learn with little preprocessing by humans. However, it is still difficult to learn stably and efficiently because the learning method usually uses a nonlinear function approximation. Neural Episodic Control (NEC), which has been proposed in order to improve s…
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.
An action selector associates, in a suitable way, to each compactly supported Hamiltonian on a symplectic manifold an action value of the Hamiltonian. Action selectors are known to exist for a broad class of symplectic manifolds. We show how the existence of an action selector leads to sharp energy capacity inequalitie…
RLMM extends psychometric models to larger tasks.
problem Sequential process data from interactive assessments are not well handled by conventional models.
method RLMM decouples person-level choice sensitivity from task-level value representation through a shared parametric action-value function.
result RLMM achieves higher estimation accuracy and lower runtime than MDP-MM in peg-solitaire simulations and AQUALAB gameplay logs.
This work improves deep reinforcement learning robustness to adversarial state uncertainty.
problem Robustness of deep reinforcement learning to adversarial state uncertainty.
method Certified adversarial robustness techniques are applied to deep reinforcement learning algorithms to compute guaranteed lower bounds on state-action values.
result The approach increases robustness to noise and adversaries in pedestrian collision avoidance and classic control tasks.
This paper investigates trajectory tracking problem for a class of underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and constrained inputs. Different from existing policy gradient methods which employ single actor-critic but cannot realize satisfactory tracking control accuracy and stable learn…
Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the enviro…