Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on th…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “baseline policies”
Algorithm safely learns from sub-optimal baseline policies while satisfying constraints.
problem Safe reinforcement learning with constraints when baseline policy is sub-optimal.
method Iterative policy optimization alternating between return maximization, baseline distance minimization, and constraint projection.
result Consistently outperforms baselines, achieving 10x fewer constraint violations and 40% higher reward.
Improves policies with high certainty, even in small samples.
problem Ensuring new policies are better than the baseline with high probability.
method Leverages powerful safety tests and multiple testing for threshold policies.
result Controls the rate of adopting a worse policy to pre-specified error level.
An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, i.e., a policy that is guaranteed to perform at least as well as a given baseline strategy. In this paper, we develop and analyze a new model-based approach to compute a safe policy when we have access …
Risk-controlled post-processing optimizes decision policies under risk constraints.
problem Optimizing decision policies with risk constraints for better outcomes.
method Developed a post-processing algorithm that selects a threshold based on fitted fallback policy and score, leveraging tools from algorithmic stability and stochastic processes.
result The post-processed policy achieves precise expected risk control under exchangeability and meets or nearly meets risk budgets while preserving more agreement with the baseline.
Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent ba…
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probability that the trained policy performs better than the behavioural policy, also called baseline in this…
This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Boo…
RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraints, normalizing flows policy generates samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps av…
AI-driven tax policies improve economic equality and productivity.
problem Lack of appropriate economic data and limited opportunity to experiment.
method Two-level deep reinforcement learning approach to learn dynamic tax policies from observational data.
result AI-driven tax policies improve the trade-off between equality and productivity by 16%.
We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system. Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking. Since the state dynamics and rewards depend on…
IPO optimizes reinforcement learning with constraints for better performance.
problem Maximizing long-term reward while satisfying cumulative constraints in decision problems.
method Interior-point Policy Optimization (IPO) using logarithmic barrier functions.
result IPO outperforms state-of-the-art baselines in reward maximization and constraint satisfaction.
This work examines the role of reinforcement learning in reducing the severity of on-road collisions by controlling velocity and steering in situations in which contact is imminent. We construct a model, given camera images as input, that is capable of learning and predicting the dynamics of obstacles, cars and pedestr…
Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …
Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to depend on both the state and action and suggest that this significantly reduces variance and improv…
MAP-Elites generates diverse trading strategies for improved execution performance.
problem Optimizing trading execution schedules in volatile market conditions.
method Quality-diversity algorithm (MAP-Elites) generating a portfolio of specialized strategies.
result Diverse strategies achieve 8-10% performance improvements, validating quality-diversity methods.
We propose an estimator and confidence interval for computing the value of a policy from off-policy data in the contextual bandit setting. To this end we apply empirical likelihood techniques to formulate our estimator and confidence interval as simple convex optimization problems. Using the lower bound of our confiden…
STAR framework reduces OPE variance by distilling complex problems into discrete ARPs.
problem High variance and bias in off-policy evaluation methods.
method STAR framework that includes various OPE estimators and leverages state abstraction.
result Predictions from ARPs estimated from off-policy data are asymptotically correct.
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcs.LG
BASIS improves LLM reasoning by sharing batchwise rollout info, reducing MSE by 69%.
problem Improving large language model reasoning with limited rollouts and batch information.
method BASIS samples only one rollout per prompt but uses batch information to improve value function estimation.
result BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++.
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what …
StepMix algorithm ensures safe exploration in reinforcement learning with near-optimal performance.
problem Conservative exploration in reinforcement learning under episode-wise constraints.
method StepMix algorithm balances exploitation and exploration while ensuring episode-wise conservative constraint.
result StepMix achieves near-optimal regret order as in the constraint-free setting.
New algorithms ensure policies perform at least as good as a baseline in reinforcement learning.
problem Learning policies that are guaranteed to perform at least as well as a baseline in reinforcement learning.
method Introduce conservative exploration for average reward and finite horizon problems, presenting two optimistic algorithms.
result Guaranteed performance of policies at least as good as a baseline, without hindering learning ability.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
A drone catches another agile drone using competitive reinforcement learning.
problem Intercepting an agile drone with another agile drone.
method Formulated as a Competitive Reinforcement Learning problem, trained with PPO, using a high-fidelity simulation environment.
result Trained policies outperform common heuristic baselines in catch rate, time to catch, and crash rate.
The paper reveals that baselines significantly impact RL algorithms' convergence.
problem Understanding the true impact of baselines on policy optimization.
method Theoretical analysis of bandit and RL problems, focusing on natural policy gradient and EXP3.
result Baselines can determine algorithm convergence, contradicting traditional optimization theory.
Second-order estimator improves continuous-time policy evaluation.
problem Estimating value surfaces from discrete data with time-inhomogeneous dynamics.
method Moment-matching coefficients for high-order generator regression.
result Second-order estimator consistently outperforms Bellman baseline.
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…
Unified framework for stable RL learning with theoretical guarantees.
problem Lack of systematic theoretical principles guiding RL post-training methods.
method Unified theoretical framework for policy-gradient estimators and optimization algorithms.
result Establishes unbiasedness, variance expressions, and convergence guarantees.
New algorithm improves online learning under performance constraints.
problem Improving performance of existing systems in various fields.
method Conservative Constrained LinUCB (CLUCB2) algorithm for contextual linear bandits.
result Empirically outperforms existing conservative bandit algorithms.
AACE learns treatment policies from EHRs using annotations to improve accuracy.
problem Learning treatment policies from multimodal EHRs with bias and inefficiency.
method Annotation-assisted coarsened effects (AACE) method.
result AACE outperforms existing methods in predicting treatment benefit from multimodal EHRs.
SPO optimizes LLMs by eliminating group-based baselines and variance issues.
problem Frequent degenerate groups and synchronization barriers in group-based policy optimization methods.
method Single-stream Policy Optimization (SPO) replaces per-group baselines with a persistent, KL-adaptive value tracker and global advantage normalization.
result SPO converges more smoothly and attains higher accuracy than GRPO, improving maj@32 by +3.4 pp across five math benchmarks.
Proposes CoPO, a new policy optimization method for competitive games.
problem Designing efficient optimization methods for competitive Markov decision processes.
method Competitive policy optimization (CoPO) approach that exploits game-theoretic nature of competitive games.
result Stable optimization, convergence to sophisticated strategies, and higher scores compared to baseline methods.
New method speeds up lifelong learning of complex tasks.
problem Catastrophic forgetting in lifelong learning.
method Directly trains lifelong function approximators via policy gradients.
result Faster convergence and better policies achieved.
This paper compares eight DRL algorithms for dynamic hedging.
problem Optimal dynamic hedging strategies using Deep Reinforcement Learning.
method Eight DRL algorithms (MCPG, PPO, DQL, DDPG) compared using a GJR-GARCH(1,1) simulated dataset.
result MCPG and PPO outperform the Black-Scholes delta hedge baseline.
VMBPO optimizes model and policy jointly using variational lower-bound.
problem Data efficiency in RL with biased simulated data.
method Formulate variational objective function, use EM, iteratively improve model and policy.
result VMBPO is more sample-efficient and robust than model-free algorithms.
New estimator uses clustering to improve off-policy evaluation accuracy.
problem Improving off-policy evaluation accuracy when logging and evaluation policies differ.
method Proposes an estimator that shares information across similar contexts using clustering.
result Clustering contexts improves estimation accuracy, especially in deficient information settings.
We propose a method for tackling catastrophic forgetting in deep reinforcement learning that is \textit{agnostic} to the timescale of changes in the distribution of experiences, does not require knowledge of task boundaries, and can adapt in \textit{continuously} changing environments. In our \textit{policy consolidati…
Evolutionary Strategies optimize hyper-parameters for off-policy learning.
problem Hyper-parameter sensitivity in off-policy learning.
method Application of Evolutionary Strategies for online hyper-parameter tuning.
result Our method outperforms state-of-the-art baselines.
Paper proposes an RL algorithm to ensure policy performance guarantees.
problem Lack of performance guarantees for RL policies compared to baselines.
method Online model-free algorithm that ensures conservative exploration.
result Regret bound of for both discrete and continuous spaces.
The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge in this scenario is how to reduce the variance of policy gradient estimates for reliable policy updates. In this paper, we combine the followi…
New method optimizes policies in non-stationary environments.
problem Optimizing policies in non-stationary, context-dependent environments.
method Two-phase approach: offline learning and online adaptation.
result Our method outperforms existing approaches in both synthetic and real-world datasets.
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
problem Critic-side gradient ill-conditioning in PPO for MTRL.
method Critic Balancing modules to improve gradient conditioning and balance task updates.
result TOPPO achieves stronger mean and tail-task performance than SAC-family and ARS-family baselines.
MetaTrader combines diverse expert strategies to optimize portfolio performance.
problem Optimizing portfolio performance in changing financial markets.
method Two-stage RL approach: imitation learning followed by a meta-policy.
result MetaTrader significantly outperforms state-of-the-art baselines in balancing profits and risks.
One-step policy improvement outperforms iterative RL methods on D4RL.
problem Improving offline RL without off-policy evaluation.
method One-step constrained/regularized policy improvement using on-policy Q estimates.
result One-step algorithm outperforms iterative algorithms on D4RL benchmark.
We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric distribution classes. We show that optimizing over such sets results in local movement in the actio…
While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex p…
Proposes a conservative exploration method for RL agents.
problem Guaranteeing performance of exploratory policies in RL.
method Importance sampling for off-policy policy evaluation.
result Derives a regret bound ensuring no conservative constraint violation.