A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New method fine-tunes discrete diffusion models for RLHF tasks.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
RL solves discrete LQ control with Gaussian optimal policy.
Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gradient based deep RL algorithms efficiently. To effectively operate in multidimensional discrete acti…
We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments. Previous work for discrete treatment/action spaces focuses on inverse probability weighting (IPW) and doubly robust (DR) methods that use a reject…
Efficient deep policy gradient method for continuous-time control problems.
Paper formulates mutual information optimal control for discrete-time systems.
Study proposes a new GAN for realistic discrete financial orders.
Hybrid Policy Optimization tackles reinforcement learning in hybrid spaces, improving performance over PPO.
D2D-SPL uses discrete states and a classifier to train RL faster.
Optimizes control of noisy discrete systems without system matrix knowledge.
Develops deep jump learning for continuous treatment OPE.
Faster policy learning via continuous-time gradients.
Discrete diffusion samplers improve sampling from unnormalised densities.
It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequ…
LHIEM model predicts health, income, and employment over years.
Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …
For an autonomous agent, executing a poor policy may be costly or even dangerous. For such agents, it is desirable to determine confidence interval lower bounds on the performance of any given policy without executing said policy. Current methods for exact high confidence off-policy evaluation that use importance sampl…
GFlowNet-EM learns complex latent variable models with discrete structures.
Study improves sampling efficiency of diffusion models using RL and PDEs.
In this paper, we propose an information-theoretic exploration strategy for stochastic, discrete multi-armed bandits that achieves optimal regret. Our strategy is based on the value of information criterion. This criterion measures the trade-off between policy information and obtainable rewards. High amounts of policy …
STAR framework reduces OPE variance by distilling complex problems into discrete ARPs.
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy …
The discrete-time mean-variance portfolio selection formulation, a representative of general dynamic mean-risk portfolio selection problems, does not satisfy time consistency in efficiency (TCIE) in general, i.e., a truncated pre-committed efficient policy may become inefficient when considering the corresponding trunc…
Policy-gradient method controls multiple non-cohesive targets.
Inventory control with unknown demand distribution is considered, with emphasis placed on the case involving discrete nonperishable items. We focus on an adaptive policy which in every period uses, as much as possible, the optimal newsvendor ordering quantity for the empirical distribution learned up to that period. Th…
Low precision networks in the reinforcement learning (RL) setting are relatively unexplored because of the limitations of binary activations for function approximation. Here, in the discrete action ATARI domain, we demonstrate, for the first time, that low precision policy distillation from a high precision network pro…
Naive investors make riskier choices than optimal strategies in continuous-time finance.
The paper tackles counterfactual learning for stochastic policies with continuous actions.
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
New insights into RL efficiency from managing time discretization.
This paper improves reinforcement learning policies in a scalable way.
Direct optimization is an appealing framework that replaces integration with optimization of a random objective for approximating gradients in models with discrete random variables. A sampling is a framework for optimizing such random objectives over large spaces. We show how to combine these techniques to yiel…
Reinforcement learning (RL) has recently been introduced to interactive recommender systems (IRS) because of its nature of learning from dynamic interactions and planning for long-run performance. As IRS is always with thousands of items to recommend (i.e., thousands of actions), most existing RL-based methods, however…
Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a st…
New approach uses deep reinforcement learning for vehicle dispatching, reducing waiting times.
New framework improves restless bandit policies for large numbers of arms.
New method combines value function decomposition and policy gradients for cooperative multi-agent reinforcement learning.
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then con…
PBVFs generalize across policies using learned value functions.
Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate. Though often achieving encouraging empirical success, its underlying mathematical principle on {\em policy-distri…
Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitiv…
Paper tackles RL with continuous actions and unmeasured confounders.
Deep reinforcement learning boosts commodities trading performance.
Paper proposes an RL algorithm to ensure policy performance guarantees.
Transfer Learning (TL) has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing transfer approaches either explicitly computes the similarity between tasks or select appropriate source policies to provide guided explorations…
Revisits PPO design choices, exposing failure modes and proposing alternatives.
We study the problem of identifying the policy space of a learning agent, having access to a set of demonstrations generated by its optimal policy. We introduce an approach based on statistical testing to identify the set of policy parameters the agent can control, within a larger parametric policy space. After present…