Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “action-values”
New approach for open ad hoc teamwork using graph-based policy learning.
problem Designing autonomous agents to collaborate with changing teams without prior coordination.
method Graph-based policy learning to adapt to dynamic team compositions.
result Successfully models the effects of other agents, leading to robust adaptation and superior performance.
A new RL algorithm POWR learns world models to estimate action-values.
problem Inaccessibility of explicit action-value functions in RL.
method Learning a world model using conditional mean embeddings and deriving action-value function via matrix operations.
result POWR algorithm converges to global optimum with proven rates.
A new method learns action representations for reinforcement learning.
problem Efficient action-value estimation in reinforcement learning.
method Action hypergraph networks framework for learning action representations.
result Hypergraph Q-networks show effectiveness on various domains.
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
Q()-Learning improves Q-Learning by separating action-value functions into different time scales.
problem Q-Learning struggles with bias-variance trade-off, especially in long-term rewards.
method Introduces Q()-Learning, extending TD() to decompose Q()-function into distinct discount factors.
result Q()-Learning achieves better stability and scalability, especially for long-term tasks.
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifte…
New method QMLE performs well in complex action spaces without policy gradients.
problem Why policy gradients outperform action-value methods in complex action spaces.
method QMLE framework for action-value methods based on three principles.
result QMLE performs comparably to policy gradient methods in complex action spaces.
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicingcs.LG
Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicin…
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
problem Overestimation and high variance in Q-Learning cause learning algorithms to diverge in complex environments.
method Deep Weighted Q-Learning (Deep WQL) uses Dropout and Monte Carlo sampling to approximate WQL's weights and reduce bias.
result Deep WQL reduces bias and improves performance on benchmarks compared to existing methods.
MAGE optimizes policies using action gradients from model-based learning.
problem Lack of direct gradient information from critics in actor-critic methods.
method Model-based actor-critic algorithm that learns action-value gradient.
result MAGE outperforms model-free and model-based baselines on continuous control tasks.
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
problem Determining the minimum number of queries needed for sound planners in MDPs with linear function approximation.
method Analyzing fixed-horizon and discounted MDPs with a generative model, showing lower bounds on the number of queries required.
result Sound planners need at least exponential number of queries in both fixed-horizon and discounted settings.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
This paper investigates learning sparse representations and action-value functions simultaneously in deep reinforcement learning.
problem Mitigating catastrophic interference and improving cumulative reward in deep reinforcement learning agents.
method Employing regularization techniques to learn sparse representations and action-value functions incrementally.
result Learning sparse representations can improve performance and robustness in deep reinforcement learning agents.
Bayesian framework for learning optimal action-value function in MDPs.
problem Uncertainty quantification in MDPs for optimal decision-making strategies.
method Full Bayesian framework including modelling, inference, and decision-making.
result Demonstrates exploration benefits of posterior sampling in MDPs.
QMIX combines per-agent values to create decentralised policies.
problem Training decentralised policies from centralised learning.
method QMIX uses a mixing network to estimate joint action-values as a monotonic combination of per-agent values.
result QMIX significantly outperforms existing methods on the StarCraft Multi-Agent Challenge (SMAC).
Transformers learn to predict chess moves with surprising accuracy and strength.
problem Training transformers on chess to predict moves accurately.
method Large-scale chess dataset (10M games), supervised learning with up to 270M parameters.
result Transformers can predict action-values for novel boards with high accuracy.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
Diffusion-QL uses diffusion models to improve offline RL performance.
problem Offline RL struggles with function approximation errors on out-of-distribution actions.
method Diffusion-QL represents the policy as a conditional diffusion model and optimizes action-values.
result Diffusion-QL achieves state-of-the-art performance on D4RL benchmark tasks.
QTRAN++ improves MARL performance in complex environments.
problem Poor empirical performance of QTRAN in complex environments.
method Stabilizing training objective, removing role separation, and introducing a multi-head mixing network.
result QTRAN++ achieves state-of-the-art performance in the Starcraft Multi-Agent Challenge (SMAC).
New algorithms speed up inverse reinforcement learning by solving MDPs once.
problem Slow convergence in Maximum Entropy Inverse Reinforcement Learning.
method Deep Inverse Q-learning with constraints exploiting Q-learning.
result Up to several orders of magnitude speedup compared to existing methods.
Latent variable models improve RL by facilitating efficient learning and exploration.
problem Improving sample efficiency in reinforcement learning.
method Representation view of latent variable models for state-action value functions, incorporating kernel embeddings and UCB exploration.
result Established sample complexity of the proposed approach in online and offline settings, demonstrated superior performance in benchmarks.
Paper presents an efficient exploration method for reinforcement learning.
problem Efficient exploration in reinforcement learning with uncertainty quantification.
method Parameterized Indexed Value Function (PIV) using index sampling.
result Proves the regret bound for learning PIV in a tabular setting and proposes PINs for computational learning.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcs.LG
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones f…
Band-limited SAC improves learning efficiency and stability in simulated environments.
problem Improving sample efficiency and stability in SAC algorithms.
method Artificially bandlimiting the target critic's spatial resolution using a convolutional filter.
result Bandlimited SAC outperforms classic twin-critic SAC in various Gym environments and is more stable.
Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…
Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on …
We propose a flexible method for estimating value functions in reinforcement learning without parametric assumptions.
problem Lack of interpretability in reinforcement learning models, especially in healthcare applications.
method Nonparametric additive model using local kernel regression and basis expansion.
result Personalized, adaptive recommendations for postoperative recovery.
New framework for modular reinforcement learning reduces sample complexity.
problem Achieving independent credit assignment in reinforcement learning.
method Defining modular credit assignment as minimizing algorithmic mutual information, introducing modularity criterion for causal analysis.
result Single-step temporal difference action-value methods meet the modularity criterion, improving sample efficiency.
SAVE combines Q-learning and MCTS with amortized value estimates for improved performance.
problem Combining model-free Q-learning and model-based MCTS for efficient learning and planning.
method SAVE uses a learned prior to guide MCTS, which estimates improved state-action values. These estimates are used to update the prior, creating a cooperative relationship between learning and search.
result SAVE achieves higher rewards with fewer training steps and strong performance with small search budgets.
Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute t…
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function;…
New RL method handles large state-action spaces with complex models.
problem Complex models and large state-action spaces in reinforcement learning.
method π-KRVI, an optimistic modification of least-squares value iteration using kernel ridge regression.
result First order-optimal regret guarantees under general settings, improving over state of the art.
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.
An action selector associates, in a suitable way, to each compactly supported Hamiltonian on a symplectic manifold an action value of the Hamiltonian. Action selectors are known to exist for a broad class of symplectic manifolds. We show how the existence of an action selector leads to sharp energy capacity inequalitie…
RLMM extends psychometric models to larger tasks.
problem Sequential process data from interactive assessments are not well handled by conventional models.
method RLMM decouples person-level choice sensitivity from task-level value representation through a shared parametric action-value function.
result RLMM achieves higher estimation accuracy and lower runtime than MDP-MM in peg-solitaire simulations and AQUALAB gameplay logs.
Study shows convex contact spheres resemble contact ellipsoids.
problem Characterizing the structure of convex contact spheres.
method Stratification by Reeb orbit periods and analysis of spectral invariants.
result Any stratum of a convex contact sphere is an integral homology sphere, and spectral invariants coincide with action values.
Mean Actor Criticstat.ML
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
Why Agentic Theorem Prover Works: A Statistical Provability Theory of Mathematical Reasoning Modelsstat.ML
Study improves understanding of why agentic theorem provers succeed.
problem Understanding which components of agentic theorem provers improve proof success.
method Statistical provability theory and finite-horizon reachability MDP model.
result Bounds provability gap and explains components' effectiveness.
We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive -functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …
UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.
problem Improving value function learning in complex reinforcement learning tasks.
method Uncertainty-aware low-rank Q-matrix estimation (UA-LQE) algorithm.
result UA-LQE selectively erases uncertain entries in Q-matrix to improve value function approximation.
We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…
Imitation by observation is an approach for learning from expert demonstrations that lack action information, such as videos. Recent approaches to this problem can be placed into two broad categories: training dynamics models that aim to predict the actions taken between states, and learning rewards or features for com…