arXiv research

Traffic actors' future motion predicted using a hybrid graph model.

Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy ($π$) and the behavior policy (b) is a major cause of instability. High variance also originates from distributional mismatch. The var…

Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.

AGAC uses an adversary to enhance exploration in complex tasks.

Neural network approximates Bayesian decision-making parameters.

New actor-critic method reduces sample complexity for reinforcement learning.

Inferring a person's goal from their behavior is an important problem in applications of AI (e.g. automated assistants, recommender systems). The workhorse model for this task is the rational actor model - this amounts to assuming that people have stable reward functions, discount the future exponentially, and construc…

DSAC improves robustness of imitation learning.

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-…

New framework improves offline RL performance without complex methods.

This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data…

Method extracts taint flows to classify Bitcoin mining pools.

Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this pa…

Autonomous driving presents one of the largest problems that the robotics and artificial intelligence communities are facing at the moment, both in terms of difficulty and potential societal impact. Self-driving vehicles (SDVs) are expected to prevent road accidents and save millions of lives while improving the liveli…

Paper provides convergence guarantees for off-policy NAC with finite sample complexity.

A new Input-Output model, called the Multi-Entity Input-Output (MEIO) model, is introduced to estimate the responsibility of entities of an ecosystem on the footprint of each other. It assumed that the ecosystem is comprised of end users, service providers, and utilities. The proposed MEIO modeling approach can be seen…

MODULE solves LfO problem with high sample efficiency and stability.

The study uses AI to optimize trading in FX markets by considering size-dependent fees and risk-aversion.

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

Our focus is on realistically modeling and forecasting dynamic networks of face-to-face contacts among individuals. Important aspects of such data that lead to problems with current methods include the tendency of the contacts to move between periods of slow and rapid changes, and the dynamic heterogeneity in the actor…

Single-timescale actor-critic finds globally optimal policy.

Behavior of systems that are functions of anticipated behavior of other systems, whose own behavior is also anticipatory but homeostatic and determined by hierarchical ordering, which changes over time, of sets of possible environments that are not co-possible, is proven to be highly non-linear and sensitively dependen…

New PAC-Bayesian approach stabilizes actor-critic learning.

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondl…

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

FORK improves model-free reinforcement learning performance.

LC-SAC tackles non-stationary dynamics in reinforcement learning.

New analysis shows actor-critic method converges efficiently in practical settings.

This work analyzes actor-critic methods for faster convergence.

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

Actor-critic algorithms converge to an ODE as data samples change dynamically.

Curiosity-driven recommendations improve personalized learning efficiency and enjoyment.

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…

Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

DG separates successes and failures by gating updates with advantage and surprisal.

New Soft Actor-Critic for discrete actions.

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy pro…

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

Proposes a new model for predicting future motion of road actors in autonomous vehicles.

Actor-Critic method achieves optimal regret for unichain MDPs.

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

Proposes a value-based method for continuous control without an actor.

Optimistic actor-critic tackles linear MDPs with parametric policies.

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.

ESAC improves reinforcement learning by lookahead and intuition.