arXiv research

New method identifies critical states to improve RL agent explainability and speed.

Enhances reinforcement learning with partial state information.

New method improves deep policy gradient algorithms by learning relative state values.

Classification of ground state solutions to critical Dirac equation on spheres.

We prove sharp pointwise decay estimates for critical Dirac equations on $\mathbb{R}^n$ with $n\geq 2$. They appear for instance in the study of critical Dirac equations on compact spin manifolds, describing blow-up profiles, and as effective equations in honeycomb structures. For the latter case, we find excited state…

ESAC improves reinforcement learning by lookahead and intuition.

Study on natural actor-critic for POMDPs with finite memory.

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.

We consider the estimation of the policy gradient in partially observable Markov decision processes (POMDP) with a special class of structured policies that are finite-state controllers. We show that the gradient estimation can be done in the Actor-Critic framework, by making the critic compute a "value" function that …

Researchers identify critical protein residues using advanced graph theory.

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

Improved exploration in RL with latent state marginalization.

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to dis…

Improved bounds for eigenfunctions on hyperbolic surfaces found.

Improved sample complexity for actor-critic algorithms in MDPs.

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

First we provide a simple set of sufficient conditions for the weak convergence of scaled affine processes with state space $R_+ \times R^d$. We specialize our result to one-dimensional continuous state branching processes with immigration. As an application, we study the asymptotic behavior of least squares estimators…

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address th…

Actor-Critic method achieves optimal regret for unichain MDPs.

We study the problem of predicting rare critical transition events for a class of slow-fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces critical transitions. By taking advantage of recent advances in reservoir…

FORK improves model-free reinforcement learning performance.

Reinforcement learning methods carry a well known bias-variance trade-off in n-step algorithms for optimal control. Unfortunately, this has rarely been addressed in current research. This trade-off principle holds independent of the choice of the algorithm, such as n-step SARSA, n-step Expected SARSA or n-step Tree bac…

Band-limited SAC improves learning efficiency and stability in simulated environments.

CARL safely adapts RL agents for safety-critical tasks.

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

Critic-regularized regression improves offline RL performance.

The Levy-Gromov inequality states that round spheres have the least isoperimetric profile (normalized by total volume) among Riemannian manifolds with a fixed positive lower bound on the Ricci tensor. In this note we study critical metrics corresponding to the Levy-Gromov inequality and prove that, in two-dimensions, t…

The study of the critical dynamics in complex systems is always interesting yet challenging. Here, we choose financial market as an example of a complex system, and do a comparative analyses of two stock markets - the S&P 500 (USA) and Nikkei 225 (JPN). Our analyses are based on the evolution of crosscorrelation struct…

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yieldi…

New PAC-Bayesian approach stabilizes actor-critic learning.

New findings on Chern flat metrics and their criticality.

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…

A new method identifies critical transitions in high-dimensional data.

A simple model economy with locally interacting producers and consumers is introduced. When driven by extremal dynamics, the model self-organizes {\em not} to an attractor state, but to an asymptote, on which the economy has a constant rate of deflation, is critical, and exhibits avalanches of activity with power-law d…

PBVFs generalize across policies using learned value functions.

We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean o…

The paper studies the smoothness of critical points of variational integrals on Hessian spaces.

Paper tackles overestimation bias in continuous control, improving performance by 25%.

DMs emerge from DenseAMs, transitioning from memorization to generalization.

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a pol…

Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world sa…

This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expect…

D2D-SPL uses discrete states and a classifier to train RL faster.

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and acti…

The Stock Market is a complex self-interacting system, characterized by an intermittent behaviour. Periods of high activity alternate with periods of relative calm. In the present work we investigate empirically about the possibility that the market is in a self-organized critical state (SOC). A wavelet transform metho…

MAGE optimizes policies using action gradients from model-based learning.

New analysis shows actor-critic method converges efficiently in practical settings.