A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
New algorithms improve policy evaluation in reinforcement learning.
This work analyzes the gap between off-policy and on-policy policy gradient methods and provides conditions to reduce this gap.
MPPO improves exploration efficiency for on-policy methods.
Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually d…
Study investigates key design choices in on-policy RL algorithms.
Adapts GRPO for off-policy RL, improving reward.
RPO uses past and future state-action info for better policy optimization.
New RL algorithm tackles complex discrete action spaces.
Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a partic…
Cramming method evaluates learned policies from contextual bandits efficiently.
New RL algorithms improve average-reward performance.
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
On-policy reinforcement learning (RL) algorithms have high sample complexity while off-policy algorithms are difficult to tune. Merging the two holds the promise to develop efficient algorithms that generalize across diverse environments. It is however challenging in practice to find suitable hyper-parameters that gove…
Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updat…
Improves reinforcement learning stability and efficiency.
Proposes log density gradient to improve reinforcement learning sample complexity.
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, an…
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
New reinforcement learning algorithms improve policy optimization with entropy regularization.
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…
Minimal assumptions analysis of Q-learning with time-varying policies.
CODA resolves coordination issues in offline multi-agent reinforcement learning.
AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.
New findings explain why online methods outperform offline methods in noisy expert feedback settings.
A highly influential ingredient of many techniques designed to exploit sparsity in numerical optimization is the so-called chordal extension of a graph representation of the optimization problem. The definitive relation between chordal extension and the performance of the optimization algorithm that uses the extension …
Develops first-order methods for average-reward MDPs with strong guarantees.
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…
Many reinforcement learning applications involve the use of data that is sensitive, such as medical records of patients or financial information. However, most current reinforcement learning methods can leak information contained within the (possibly sensitive) data on which they are trained. To address this problem, w…
Policy gradient aims to maximize expected return using gradient ascent.
This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
Large deviations theory applied to policy gradient methods.
Bayesian method infers contextual bandit policies robustly.
We study reinforcement learning of chatbots with recurrent neural network architectures when the rewards are noisy and expensive to obtain. For instance, a chatbot used in automated customer service support can be scored by quality assurance agents, but this process can be expensive, time consuming and noisy. Previous …
New RL algorithms improve control tasks with data reuse.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evalua…
We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies…
Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this p…
Mirror descent method improved RL algorithms.
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use param…
Efficient exploration remains a challenging research problem in reinforcement learning, especially when an environment contains large state spaces, deceptive local optima, or sparse rewards. To tackle this problem, we present a diversity-driven approach for exploration, which can be easily combined with both off- and o…
Transformer improves sequence generation with insertion and deletion phases.
Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.
Several approximate policy iteration schemes without value functions, which focus on policy representation using classifiers and address policy learning as a supervised learning problem, have been proposed recently. Finding good policies with such methods requires not only an appropriate classifier, but also reliable e…
Policy analysts wish to visualize a range of policies for large simulator-defined Markov Decision Processes (MDPs). One visualization approach is to invoke the simulator to generate on-policy trajectories and then visualize those trajectories. When the simulator is expensive, this is not practical, and some method is r…
Sparse reward is one of the biggest challenges in reinforcement learning (RL). In this paper, we propose a novel method called Generative Exploration and Exploitation (GENE) to overcome sparse reward. GENE automatically generates start states to encourage the agent to explore the environment and to exploit received rew…