Proposes a conservative exploration method for RL agents.
problem Guaranteeing performance of exploratory policies in RL.
method Importance sampling for off-policy policy evaluation.
result Derives a regret bound ensuring no conservative constraint violation.
Theoretical analysis confirms non-conservative algorithms can converge to optimal policies.
problem Theoretical guarantees for non-conservative reinforcement learning algorithms.
method Theoretical analysis of Peng's Q( λ λ λ ) algorithm. result Peng's Q( λ λ λ ) converges to an optimal policy under certain conditions. BCPO optimizes offline RL policies by converting uncertainty into conservative bounds.
problem Offline RL's fragility under distribution shifts and model errors.
method Bayesian approach with credible lower bounds and KL regularization.
result BCPO yields an uncertainty-calibrated policy that avoids exploiting model errors.
HAMBO estimates policy performance by hallucinating worst-case trajectories, providing valid lower bounds.
problem Conservative off-policy evaluation of policies in real-world applications.
method HAMBO hallucinates worst-case trajectories based on learned model uncertainty.
result Valid lower bounds on policy performance, converging to true expected return under regular conditions.
New algorithms ensure policies perform at least as good as a baseline in reinforcement learning.
problem Learning policies that are guaranteed to perform at least as well as a baseline in reinforcement learning.
method Introduce conservative exploration for average reward and finite horizon problems, presenting two optimistic algorithms.
result Guaranteed performance of policies at least as good as a baseline, without hindering learning ability.
Paper proposes an RL algorithm to ensure policy performance guarantees.
problem Lack of performance guarantees for RL policies compared to baselines.
method Online model-free algorithm that ensures conservative exploration.
result Regret bound of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) for both discrete and continuous spaces. StepMix algorithm ensures safe exploration in reinforcement learning with near-optimal performance.
problem Conservative exploration in reinforcement learning under episode-wise constraints.
method StepMix algorithm balances exploitation and exploration while ensuring episode-wise conservative constraint.
result StepMix achieves near-optimal regret order as in the constraint-free setting.
CQL learns conservative Q-functions to improve offline RL performance.
problem Leveraging large, static datasets in reinforcement learning without further interaction.
method Conservative Q-learning (CQL) which learns a conservative Q-function to lower-bound policy values.
result CQL substantially outperforms existing offline RL methods, often achieving 2-5 times higher final returns.
VaR-CPO optimizes VaR-constrained RL problems with conservative policy updates.
problem Optimizing VaR-constrained reinforcement learning problems.
method Combines Cantelli's inequality and trust-region framework for efficient and conservative optimization.
result Achieves zero constraint violations during training in feasible environments.
SCPO learns robust policies without modeling disturbance, improving real-world task performance.
problem Poor performance of reinforcement learning in real-world tasks due to disturbance in transition dynamics.
method State-conservative policy optimization (SCPO) that reduces disturbance to state space and approximates it with a gradient-based regularizer.
result SCPO learns robust policies without prior knowledge of disturbance or simulators, improving performance in robot control tasks.
The paper tackles safe exploration in RL by a conservative safety critic.
problem Safe exploration in reinforcement learning (RL) when partially trained policies are deployed.
method Learning a conservative safety estimate through a critic, provably bounding catastrophic failures.
result The approach provably converges to competitive task performance with significantly lower catastrophic failure rates.
Canary optimizes VaR-constrained RL problems with a conservative bound using Cantelli's inequality.
problem Optimizing reinforcement learning policies under VaR constraints in dense cost regimes.
method Employing Cantelli's inequality to create a conservative and smooth bound on VaR constraints based on moments of cost returns. Extending trust-region framework for worst-case bounds on policy improvement and constraint violation.
result Canary reliably satisfies VaR constraints with fewest violations and earliest permanent satisfaction, while maintaining reward competitiveness.
RORL improves offline RL robustness with conservative smoothing.
problem Distribution shift and robustness issues in offline RL.
method RORL introduces regularization and conservative smoothing for robustness.
result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.
New algorithm improves online learning under performance constraints.
problem Improving performance of existing systems in various fields.
method Conservative Constrained LinUCB (CLUCB2) algorithm for contextual linear bandits.
result Empirically outperforms existing conservative bandit algorithms.
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservati…
New batch RL method avoids overly optimistic policies.
problem Challenges in applying RL to large domains with limited data.
method Modified Bellman optimality and evaluation back-up for conservative updates.
result Can find approximately best policies within explored state-action space.
Paper presents a reduction-based framework for conservative bandits and RL with improved lower and upper bounds.
problem Conservative bandits and reinforcement learning problems.
method Reduction technique to calculate necessary and sufficient budget from baseline policy.
result Improved lower and upper bounds for various conservative settings.
Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this pa…
Proposes a method to avoid excessive exploration in reinforcement learning.
problem Avoiding excessive exploration in reinforcement learning to deploy it in practice.
method Designs a novel algorithm using UCB reinforcement learning policy with adaptive exploration constraints.
result Proves that the approach remains conservative while minimizing regret in tabular settings and validates on real-world tasks.
Conservative Policy Iteration (CPI) is a founding algorithm of Approximate Dynamic Programming (ADP). Its core principle is to stabilize greediness through stochastic mixtures of consecutive policies. It comes with strong theoretical guarantees, and inspired approaches in deep Reinforcement Learning (RL). However, CPI …
Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
Higher conservative training increases reward-hacking in reasoning models.
problem Reward hacking during online adaptation in reasoning models.
method Conservative offline training with varying levels of conservatism (β) was applied to a Qwen3-14B policy, and online adaptation was measured against a reward ensemble.
result Higher conservatism (β) increases reward-hacking damage, measured by the Goodhart gap and AUGC.
POMBU improves model-based RL's asymptotic performance by estimating and using uncertainty.
problem Model-based reinforcement learning struggles with model errors, leading to suboptimal performance.
method POMBU uses estimated uncertainty to optimize policies conservatively, improving asymptotic performance.
result POMBU outperforms existing methods in sample efficiency and asymptotic performance.
Enhances deep RL learning with stable policy updates.
problem Stability and performance degradation in deep RL methods.
method EVEREST method providing conservative updates with confidence bounds.
result Significant improvements in continuous control and Atari benchmarks.
Proposes a method to improve few-shot transfer in off-dynamics RL.
problem Traditional RL struggles with transferring policies between environments with different dynamics.
method Introduces a penalty to regulate source-trained policies in target environments with limited data.
result Improves performance in various off-dynamics RL scenarios compared to existing methods.
New algorithms optimize a soft-robust criterion in reinforcement learning, reducing conservatism.
problem Computing robust policies for high-stakes decisions with limited data.
method Soft-robust criterion using risk measures, two algorithms for optimization.
result Our algorithms produce less conservative solutions than existing methods.
Develops a support-aware framework for reserve-policy selection in advertising markets.
problem Log-based reserve-price evaluation risks weak support and subgroup harm.
method Support-aware offline decision framework converting logged evidence into certified policies.
result Preserves the best gate-passing policy while eliminating only policies with certified regret.
Develops algorithms for CCBs with non-linear costs, improving safety and performance.
problem Safety constraints in sequential decision making with non-linear arm costs.
method Innovative algorithms using Inverse Gap Weighting (IGW) and online regression oracle.
result Sub-linear regret bounds for C-SquareCB and first-order regret for C-FastCB.
Paper proposes a new DRL algorithm optimizing Spectral Risk Measures for better risk management.
problem Inconsistencies and conservatism in existing risk measures in DRL.
method Optimizes a broader class of static Spectral Risk Measures (SRM) in DRL.
result Demonstrates improved performance over existing risk-neutral and risk-sensitive DRL models.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
BRPO optimizes batch RL policies to better exploit state-action differences.
problem Batch RL's conservatism limits exploitation of state-action differences.
method Proposes residual policies and derives BRPO to maximize policy performance.
result BRPO achieves state-of-the-art performance in various tasks.
New offline RL algorithms tackle partial data coverage with optimal performance and practicality.
problem Partial data coverage in offline RL datasets.
method Augmented Lagrangian method applied to MIS formulation for optimal offline RL.
result Statistically optimal offline RL with practical performance, eliminating conservatism.
New bounds assess policy evaluation under unobserved confounders, showing model-based methods are more effective.
problem Policy evaluation under unobserved confounders in uncertain causal environments.
method Developed worst-case bounds for sensitivity to unobserved confounders, demonstrating model-based methods are more effective.
result Model-based approaches with robust MDPs provide sharper lower bounds for policy evaluation.
EcoCast predicts biodiversity risks using satellite data and citizen science records.
problem Unprecedented shifts in species distributions due to climate change and habitat loss.
method Spatio-temporal model using sequence-based transformers and continual learning.
result Promising improvements in forecasting bird species distributions compared to Random Forest.
A new framework estimates policy value robustly against confounders.
problem Estimating policy value in the presence of unobserved confounders.
method Convex programming for sharp lower bounds.
result Sharp lower bounds on policy value for robust inference.
Robustness is important for sequential decision making in a stochastic dynamic environment with uncertain probabilistic parameters. We address the problem of using robust MDPs (RMDPs) to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution is det…
Adapts safe policies for exploration in high-risk settings.
problem Balancing safety and exploration in high-risk environments.
method Uses conformal calibration on a safe reference policy to determine aggressive action limits.
result Safe exploration improves performance without requiring model class identification or hyperparameter tuning.
Study maps research streams in biodiversity finance, identifies key areas.
problem Biodiversity loss and need for finance to reverse trends.
method Quantitative bibliometric analysis of 189,456 references.
result Identifies eight primary research streams in biodiversity finance.
Paper analyzes convergence of entropy-regularized reinforcement learning methods.
problem Ensuring sub-optimality control in entropy-regularized RL.
method Unified analysis of regularized MPI and VI schemes, providing convergence rates.
result Established sufficient conditions for convergence and provided explicit rates.
New RL algorithms learn policies competitive with best in class without assuming optimal policy.
problem Agnostic RL where best policy not necessarily optimal.
method First-order optimization in non-Euclidean space, reducing to policy learning.
result Sample complexity upper bounds for three algorithms under VGD condition.
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probability that the trained policy performs better than the behavioural policy, also called baseline in this…
MoMA improves model-based RL by using unrestricted policy classes.
problem Limited sample efficiency and generalizability in model-based offline RL.
method Model-based mirror ascent algorithm with general function approximations.
result Theoretical guarantees and practical implementation of MoMA.
We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe p…
Excessively changing policies in many real world scenarios is difficult, unethical, or expensive. After all, doctor guidelines, tax codes, and price lists can only be reprinted so often. We may thus want to only change a policy when it is probable that the change is beneficial. In cases that a policy is a threshold on …
A new method learns robust policies from offline data with latent structures.
problem Conservative policies under unrealistic dynamics shifts.
method d-RRMDP framework with f f f -divergence regularization and R2PVI algorithm. result R2PVI learns robust policies with superior computational efficiency.
Paper connects DP and optimization for RL, suggesting new algorithms.
problem Optimizing scalar objectives in RL.
method Drawing connections between DP and optimization algorithms.
result Links between DP schemes and optimization algorithms.
AdaRL improves robust RL by adaptively adjusting policy complexity.
problem Handling epistemic uncertainty in environment dynamics.
method Bi-level optimization framework with adaptive rank adjustment.
result AdaRL outperforms existing methods on MuJoCo benchmarks.
Combines multiple OPE estimators into a more accurate and efficient estimate.
problem Offline evaluation of recommender systems using biased data.
method Meta-analysis of correlated OPE estimators, accounting for inter-estimator correlation.
result Improved statistical efficiency and accuracy in estimating policy value.