Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Action-Value Critic”
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
Band-limited SAC improves learning efficiency and stability in simulated environments.
problem Improving sample efficiency and stability in SAC algorithms.
method Artificially bandlimiting the target critic's spatial resolution using a convolutional filter.
result Bandlimited SAC outperforms classic twin-critic SAC in various Gym environments and is more stable.
A new method learns action representations for reinforcement learning.
problem Efficient action-value estimation in reinforcement learning.
method Action hypergraph networks framework for learning action representations.
result Hypergraph Q-networks show effectiveness on various domains.
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
Mean Actor Criticstat.ML
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
This paper investigates trajectory tracking problem for a class of underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and constrained inputs. Different from existing policy gradient methods which employ single actor-critic but cannot realize satisfactory tracking control accuracy and stable learn…
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Costcs.LG
Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…
Paper introduces a meta-critic for accelerating off-policy actor-critic learning.
problem Improving sample efficiency in continuous control tasks.
method Meta-critic that meta-learns an additional loss for the actor.
result Online meta-critic learning leads to improved performance in various continuous control environments.
New approach for open ad hoc teamwork using graph-based policy learning.
problem Designing autonomous agents to collaborate with changing teams without prior coordination.
method Graph-based policy learning to adapt to dynamic team compositions.
result Successfully models the effects of other agents, leading to robust adaptation and superior performance.
New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
A new RL algorithm POWR learns world models to estimate action-values.
problem Inaccessibility of explicit action-value functions in RL.
method Learning a world model using conditional mean embeddings and deriving action-value function via matrix operations.
result POWR algorithm converges to global optimum with proven rates.
We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy pro…
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
Optimistic Actor-Critic improves exploration efficiency in reinforcement learning.
problem Poor sample efficiency in existing actor-critic methods.
method Introduces Optimistic Actor-Critic, approximating upper and lower bounds on state-action value function.
result Achieves state-of-the-art sample efficiency in challenging continuous control tasks.
Q()-Learning improves Q-Learning by separating action-value functions into different time scales.
problem Q-Learning struggles with bias-variance trade-off, especially in long-term rewards.
method Introduces Q()-Learning, extending TD() to decompose Q()-function into distinct discount factors.
result Q()-Learning achieves better stability and scalability, especially for long-term tasks.
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifte…
FACMAC combines deep policy gradients with factored critic for multi-agent reinforcement learning.
problem Cooperative multi-agent reinforcement learning in discrete and continuous action spaces.
method FACMAC uses a centralised but factored critic, combining per-agent utilities into a joint action-value function.
result FACMAC outperforms MADDPG and other baselines on multi-agent particle environments and StarCraft II tasks.
We consider tackling a single-agent RL problem by distributing it to learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local plan…
RLMM extends psychometric models to larger tasks.
problem Sequential process data from interactive assessments are not well handled by conventional models.
method RLMM decouples person-level choice sensitivity from task-level value representation through a shared parametric action-value function.
result RLMM achieves higher estimation accuracy and lower runtime than MDP-MM in peg-solitaire simulations and AQUALAB gameplay logs.
LICA learns credit assignment for cooperative agents without explicit formulation.
problem Credit assignment among cooperative agents.
method Centralized critic as hypernetwork, adaptive entropy regularization.
result Significantly outperforms previous methods on benchmarks.
New method QMLE performs well in complex action spaces without policy gradients.
problem Why policy gradients outperform action-value methods in complex action spaces.
method QMLE framework for action-value methods based on three principles.
result QMLE performs comparably to policy gradient methods in complex action spaces.
Decoupled Q-Chunkingcs.LG
Decouples critic chunk length from policy to improve policy reactivity and performance.
problem Bootstrapping bias and difficulty in extracting optimal policies from chunked critics.
method Optimizes policy against a distilled critic for partial action chunks, allowing shorter chunks for policy.
result Reliably outperforms prior methods on long-horizon offline goal-conditioned tasks.
Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in…
Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy () and the behavior policy (b) is a major cause of instability. High variance also originates from distributional mismatch. The var…
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicingcs.LG
Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicin…
Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Criticcs.LG
Improves RL generalization by minimizing adversarial risk.
problem Overfitting to training environments and poor generalization to unseen scenarios.
method Introduces minimax formulation and distributional framework to RL.
result Trained policy shows improved generalization to different environments.
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
problem Overestimation and high variance in Q-Learning cause learning algorithms to diverge in complex environments.
method Deep Weighted Q-Learning (Deep WQL) uses Dropout and Monte Carlo sampling to approximate WQL's weights and reduce bias.
result Deep WQL reduces bias and improves performance on benchmarks compared to existing methods.
Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
problem Determining the minimum number of queries needed for sound planners in MDPs with linear function approximation.
method Analyzing fixed-horizon and discounted MDPs with a generative model, showing lower bounds on the number of queries required.
result Sound planners need at least exponential number of queries in both fixed-horizon and discounted settings.
This work improves deep reinforcement learning robustness to adversarial state uncertainty.
problem Robustness of deep reinforcement learning to adversarial state uncertainty.
method Certified adversarial robustness techniques are applied to deep reinforcement learning algorithms to compute guaranteed lower bounds on state-action values.
result The approach increases robustness to noise and adversaries in pedestrian collision avoidance and classic control tasks.
To address the challenge of backpropagating the gradient through categorical variables, we propose the augment-REINFORCE-swap-merge (ARSM) gradient estimator that is unbiased and has low variance. ARSM first uses variable augmentation, REINFORCE, and Rao-Blackwellization to re-express the gradient as an expectation und…
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
This paper investigates learning sparse representations and action-value functions simultaneously in deep reinforcement learning.
problem Mitigating catastrophic interference and improving cumulative reward in deep reinforcement learning agents.
method Employing regularization techniques to learn sparse representations and action-value functions incrementally.
result Learning sparse representations can improve performance and robustness in deep reinforcement learning agents.
Bayesian framework for learning optimal action-value function in MDPs.
problem Uncertainty quantification in MDPs for optimal decision-making strategies.
method Full Bayesian framework including modelling, inference, and decision-making.
result Demonstrates exploration benefits of posterior sampling in MDPs.
QMIX combines per-agent values to create decentralised policies.
problem Training decentralised policies from centralised learning.
method QMIX uses a mixing network to estimate joint action-values as a monotonic combination of per-agent values.
result QMIX significantly outperforms existing methods on the StarCraft Multi-Agent Challenge (SMAC).
Deep RBVFs improve continuous control in RL.
problem Challenges in finding optimal actions for continuous actions in RL.
method Introduced deep radial-basis value functions (RBVFs) for continuous control.
result RBF-DQN significantly outperforms value-function-only baselines and is competitive with actor-critic algorithms.
Transformers learn to predict chess moves with surprising accuracy and strength.
problem Training transformers on chess to predict moves accurately.
method Large-scale chess dataset (10M games), supervised learning with up to 270M parameters.
result Transformers can predict action-values for novel boards with high accuracy.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
Diffusion-QL uses diffusion models to improve offline RL performance.
problem Offline RL struggles with function approximation errors on out-of-distribution actions.
method Diffusion-QL represents the policy as a conditional diffusion model and optimizes action-values.
result Diffusion-QL achieves state-of-the-art performance on D4RL benchmark tasks.
New algorithms speed up inverse reinforcement learning by solving MDPs once.
problem Slow convergence in Maximum Entropy Inverse Reinforcement Learning.
method Deep Inverse Q-learning with constraints exploiting Q-learning.
result Up to several orders of magnitude speedup compared to existing methods.
QTRAN++ improves MARL performance in complex environments.
problem Poor empirical performance of QTRAN in complex environments.
method Stabilizing training objective, removing role separation, and introducing a multi-head mixing network.
result QTRAN++ achieves state-of-the-art performance in the Starcraft Multi-Agent Challenge (SMAC).
Latent variable models improve RL by facilitating efficient learning and exploration.
problem Improving sample efficiency in reinforcement learning.
method Representation view of latent variable models for state-action value functions, incorporating kernel embeddings and UCB exploration.
result Established sample complexity of the proposed approach in online and offline settings, demonstrated superior performance in benchmarks.