A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
BRPO optimizes batch RL policies to better exploit state-action differences.
The paper tackles robust reinforcement learning with performance guarantees.
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization …
New algorithms improve policy evaluation in reinforcement learning.
Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guarant…
PBVFs generalize across policies using learned value functions.
New method optimizes treatment policies to avoid winner's curse.
RPI combines imitation and reinforcement learning to improve policies efficiently.
Framework learns robust control policies from expert demonstrations.
New algorithm stabilizes RL policy learning through divergence regularization.
Extends OPE to evaluate policies using diverse logging data.
Counterfactual policy evaluation improves autonomous driving policies' generalization.
This work improves policy evaluation and selection using logarithmic smoothing for pessimistic off-policy estimation.
Simple policy search outperforms advanced learnable test-time augmentation techniques.
A new imitation learning method uses random search for simple policies, outperforming complex models.
New algorithm for sequential off-policy learning improves performance over batch methods.
New approach for off-policy learning in contextual bandits with performance guarantees.
We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …
Diffusion-QL uses diffusion models to improve offline RL performance.
RRPI improves offline RL by optimizing policies against worst-case dynamics.
Unified framework for policy learning using weak supervision.
Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of…
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
Adapts GRPO for off-policy RL, improving reward.
In the field of reinforcement learning there has been recent progress towards safety and high-confidence bounds on policy performance. However, to our knowledge, no practical methods exist for determining high-confidence policy performance bounds in the inverse reinforcement learning setting---where the true reward fun…
Proposes a conservative exploration method for RL agents.
We address the challenge of effective exploration while maintaining good performance in policy gradient methods. As a solution, we propose diverse exploration (DE) via conjugate policies. DE learns and deploys a set of conjugate policies which can be conveniently generated as a byproduct of conjugate gradient descent. …
A new multi-agent learning method improves performance in complex games.
This work explains why online imitation learning improves faster than theory predicts.
Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is n…
Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters. It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of traj…
Entropy regularization improves policy optimization in reinforcement learning.
We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents polic…
Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves da…
Enhances RL performance with a population-guided parallel learning scheme.
MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.
One of the questions that arises when designing models that learn to solve multiple tasks simultaneously is how much of the available training budget should be devoted to each individual task. We refer to any formalized approach to addressing this problem (learned or otherwise) as a task selection policy. In this work …
One-step policy improvement outperforms iterative RL methods on D4RL.
Develops ODRPO to improve RL algorithms with better performance and stability.
Off-policy deep reinforcement learning (RL) algorithms are incapable of learning solely from batch offline data without online interactions with the environment, due to the phenomenon known as \textit{extrapolation error}. This is often due to past data available in the replay buffer that may be quite different from th…
Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world ro…
Bayesian design improves by reducing policy training cost.
In this work, we provide theoretical guarantees for reward decomposition in deterministic MDPs. Reward decomposition is a special case of Hierarchical Reinforcement Learning, that allows one to learn many policies in parallel and combine them into a composite solution. Our approach builds on mapping this problem into a…
Model-based reinforcement learning algorithms tend to achieve higher sample efficiency than model-free methods. However, due to the inevitable errors of learned models, model-based methods struggle to achieve the same asymptotic performance as model-free methods. In this paper, We propose a Policy Optimization method w…
In this paper, we propose a novel framework for approximating the explicit MPC law for linear parameter-varying systems using supervised learning. In contrast to most existing approaches, we not only learn the control policy, but also a "certificate policy", that allows us to estimate the sub-optimality of the learned …
Taylor expansions improve reinforcement learning policies.
Improved RCPs for MABs using normalized weight functions.
Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stoc…