A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Study investigates key design choices in on-policy RL algorithms.
New algorithms improve policy evaluation in reinforcement learning.
Adapts GRPO for off-policy RL, improving reward.
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
This work analyzes the gap between off-policy and on-policy policy gradient methods and provides conditions to reduce this gap.
Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gradient based deep RL algorithms efficiently. To effectively operate in multidimensional discrete acti…
Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually d…
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
RPO uses past and future state-action info for better policy optimization.
On-policy reinforcement learning (RL) algorithms have high sample complexity while off-policy algorithms are difficult to tune. Merging the two holds the promise to develop efficient algorithms that generalize across diverse environments. It is however challenging in practice to find suitable hyper-parameters that gove…
Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a partic…
New RL algorithms improve average-reward performance.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
Cramming method evaluates learned policies from contextual bandits efficiently.
CODA resolves coordination issues in offline multi-agent reinforcement learning.
The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The …
A highly influential ingredient of many techniques designed to exploit sparsity in numerical optimization is the so-called chordal extension of a graph representation of the optimization problem. The definitive relation between chordal extension and the performance of the optimization algorithm that uses the extension …
Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updat…
Improves reinforcement learning stability and efficiency.
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, an…
Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy realm. We propose the soft policy gradient theorem (SPGT) for on-policy maximum entropy reinforcemen…
Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous exploration methods either rely on complex structure to estimate the novelty of states, or incur sens…
Proposes log density gradient to improve reinforcement learning sample complexity.
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…
Minimal assumptions analysis of Q-learning with time-varying policies.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.
New findings explain why online methods outperform offline methods in noisy expert feedback settings.
Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch. Meta-RL aims to address this challenge by leveraging experience from previous tasks so as to more quickly solve new tasks. However, in practice, th…
Deep reinforcement learning (DRL) on Markov decision processes (MDPs) with continuous action spaces is often approached by directly training parametric policies along the direction of estimated policy gradients (PGs). Previous research revealed that the performance of these PG algorithms depends heavily on the bias-var…
Study evaluates reinforcement learning for trading diverse stocks, finds Q-learning outperforms.
SARSA is an on-policy algorithm to learn a Markov decision process policy in reinforcement learning. We investigate the SARSA algorithm with linear function approximation under the non-i.i.d.\ data, where a single sample trajectory is available. With a Lipschitz continuous policy improvement operator that is smooth eno…
Study optimizes compute usage for LLM web agents, improving performance.
New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
Study on policy gradient for stochastic bandits using diffusion approximation.
RL agents outperform baselines in asset allocation.
This paper improves Q-learning bounds using reference-advantage decomposition.
Many reinforcement learning applications involve the use of data that is sensitive, such as medical records of patients or financial information. However, most current reinforcement learning methods can leak information contained within the (possibly sensitive) data on which they are trained. To address this problem, w…
Develops first-order methods for average-reward MDPs with strong guarantees.
Large deviations theory applied to policy gradient methods.
New RL algorithms improve control tasks with data reuse.
Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.
Study shows TD(0) with linear approx. converges for reversible Markov chains.
Proposes a Riemannian optimization for policy improvement in MDPs.
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…
Mirror descent method improved RL algorithms.