A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper proposes a policy gradient method for confounded POMDPs.
problem Estimating policy gradients for confounded POMDPs with continuous state and observation spaces.
method Developed a novel identification result to estimate policy gradients using offline data, solved conditional moment restrictions, and applied min-max learning with function approximation.
result Showed global convergence of the proposed algorithm in finding the optimal policy.
We address the problem of finding an optimal policy in a Markov decision process under a restricted policy class defined by the convex hull of a set of base policies. This problem is of great interest in applications in which a number of reasonably good (or safe) policies are already known and we are only interested in…
Revel tackles safe exploration in RL with verified symbolic policies.
problem Computational infeasibility of verifying neural networks in RL learning loops.
method Two policy classes: neurosymbolic with approximate gradients and symbolic policies for efficient verification. Mirror descent over policies to safely update and project policies.
result Revel discovers policies that outperform prior approaches to verified exploration.
New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces. However, little is known about even their most basic theoretical convergence properties, including: if and how fast they converge to a globally optimal solution or how they …
We study the problem of offline policy optimization in stochastic contextual bandit problems, where the goal is to learn a near-optimal policy based on a dataset of decision data collected by a suboptimal behavior policy. Rather than making any structural assumptions on the reward function, we assume access to a given …
In many settings, a decision-maker wishes to learn a rule, or policy, that maps from observable characteristics of an individual to an action. Examples include selecting offers, prices, advertisements, or emails to send to consumers, as well as the problem of determining which medication to prescribe to a patient. Whil…
Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learn…
This paper investigates a class of attacks targeting the confidentiality aspect of security in Deep Reinforcement Learning (DRL) policies. Recent research have established the vulnerability of supervised machine learning models (e.g., classifiers) to model extraction attacks. Such attacks leverage the loosely-restricte…
We consider the generic approach of using an experience memory to help exploration by adapting a restart distribution. That is, given the capacity to reset the state with those corresponding to the agent's past observations, we help exploration by promoting faster state-space coverage via restarting the agent from a mo…
New MARL algorithms resolve the curse of multiagency with function approximation.
problem Challenges in Multi-Agent Reinforcement Learning (MARL) due to the curse of multiagency.
method V-Learning with Policy Replay and Decentralized Optimistic Policy Mirror Descent.
result First polynomial sample complexity results for learning approximate Coarse Correlated Equilibria (CCEs) of Markov Games under decentralized linear function approximation.
We study a model of a corporation which has the possibility to choose various production/business policies with different expected profits and risks. In the model there are restrictions on the dividend distribution rates as well as restrictions on the risk the company can undertake. The objective is to maximize the exp…
We study the problem of learning policy of an infinite-horizon, discounted cost, Markov decision process (MDP) with a large number of states. We compute the actions of a policy that is nearly as good as a policy chosen by a suitable oracle from a given mixture policy class characterized by the convex hull of a set of k…
Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restr…
This paper considers the optimal dividend payment problem in piecewise-deterministic compound Poisson risk models. The objective is to maximize the expected discounted dividend payout up to the time of ruin. We provide a comparative study in this general framework of both restricted and unrestricted payment schemes, wh…
A new method recovers rewards from behavior policies using classification and regression.
problem Recovering meaningful rewards from observed behavior in reinforcement learning.
method GenPQR, a modular procedure that estimates behavior policy, evaluates soft Q-function, and recovers normalized reward using classification and regression.
result GenPQR matches or improves reward recovery compared to DeepPQR, while being simpler and more modular.
This paper studies the deviations of the regret in a stochastic multi-armed bandit problem. When the total number of plays n is known beforehand by the agent, Audibert et al. (2009) exhibit a policy such that with probability at least 1-1/n, the regret of the policy is of order log(n). They have also shown that such a …
Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional c…
Batch Reinforcement Learning (RL) algorithms attempt to choose a policy from a designer-provided class of policies given a fixed set of training data. Choosing the policy which maximizes an estimate of return often leads to over-fitting when only limited data is available, due to the size of the policy class in relatio…
Many reinforcement learning applications involve the use of data that is sensitive, such as medical records of patients or financial information. However, most current reinforcement learning methods can leak information contained within the (possibly sensitive) data on which they are trained. To address this problem, w…
We study numerical restricted volumes of (1,1) classes on compact Kahler manifolds, as introduced by Boucksom. Inspired by work of Ein-Lazarsfeld-Mustata-Nakamaye-Popa on restricted volumes of line bundles on projective manifolds, we pose a natural conjecture to the effect that irreducible components of the non-Kahler …
We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gradients. We empirically show that, despite its simplicity in implementation, entropy regularization c…