arXiv research

New method optimizes policies in non-stationary environments.

We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy …

NVMDP framework tackles non-stationary MDPs with varying discount rates.

New method stabilizes FQE by reweighting Bellman targets.

PROPO tackles non-stationary MDPs with efficient policy optimization.

New RL algorithm tackles non-stationary environments with flexible policy updates.

Estimates and infers multi-stage stationary treatment policies with variable selection.

New method improves stability of soft FQI for offline RL.

Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even for simple control problems solvable by standard dynamic programming techniques, policy gradient algorithms face non-convex optimization pro…

Prognosticator improves performance in non-stationary MDPs.

Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at …

A new method optimizes in nonstationary environments with many arms efficiently.

Improved API to achieve optimal error bound and query complexity in local planning.

New policy for non-stationary multi-armed bandits with recovering rewards.

Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.

New algorithms reduce dynamic regret in non-stationary RL environments.

GenDICE estimates stationary values from offline data.

In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power co…

Study on learning strategies in adaptive Markov games with policy regret as metric.

LC-SAC tackles non-stationary dynamics in reinforcement learning.

Develops RL algorithm for lifelong non-stationary environments.

Off-policy policy estimators that use importance sampling (IS) can suffer from high variance in long-horizon domains, and there has been particular excitement over new IS methods that leverage the structure of Markov decision processes. We analyze the variance of the most popular approaches through the viewpoint of con…

The paper improves QD policy ensembles using distribution ratio estimators.

Study on inventory control with changing demand, proposing adaptive algorithms.

A decentralized policy achieves logarithmic regret for multi-agent MAB problems with communication constraints.

A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…

New algorithm reduces decision switching in dynamic environments.

This paper proposes non-stationary factor models for financial stress in the UK.

New method estimates off-policy data without needing known behavior policy.

Develops a method to estimate policy values robustly in the presence of confounding variables.

In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correct…

MPC outperforms reactive budgeting in non-stationary return environments.

Paper uses RL for market making, improving stability in non-stationary markets.

Unified DICE estimators as regularized Lagrangians for improved off-policy evaluation.

We extend Bayesian multi-armed bandit (MAB) algorithms beyond their original setting by making use of sequential Monte Carlo (SMC) methods. A MAB is a sequential decision making problem where the goal is to learn a policy that maximizes long term payoff, where only the reward of the executed action is observed. In the …

Study tackles OPE in confounded settings, estimating policy value from proxies.

We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon pr…

Study nonparametric estimator for Markov chain transition matrices in offline setting.

We consider a non-stationary variant of a sequential stochastic optimization problem, in which the underlying cost functions may change along the horizon. We propose a measure, termed variation budget, that controls the extent of said change, and study how restrictions on this budget impact achievable performance. We i…

Algorithm reduces regret in SSP problems with LFA.

We consider an agent who is involved in a Markov decision process and receives a vector of outcomes every round. Her objective is to maximize a global concave reward function on the average vectorial outcome. The problem models applications such as multi-objective optimization, maximum entropy exploration, and constrai…

The paper studies risk-sensitive MDPs with recursive risk measures.

Novel approach models opponent learning dynamics in multi-agent reinforcement learning.

Novel RL-based NPG improves multi-objective NAS efficiency and performance.

New algorithm optimizes resource allocation in non-stationary networks.

The multi-armed bandit problem has been extensively studied under the stationary assumption. However in reality, this assumption often does not hold because the distributions of rewards themselves may change over time. In this paper, we propose a change-detection (CD) based framework for multi-armed bandit problems und…

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…

Optimizes consumption under regime-switching economic states with risk-sensitive preferences.