arXiv research

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

New Bellman error estimator improves offline model selection performance.

Paper studies offline RL with linear approx, focusing on inherent Bellman error.

New method stabilizes FQE by reweighting Bellman targets.

A new method calibrates value predictions in offline RL to improve reliability.

Two new algorithms improve Q* approximation in batch RL with linear error propagation.

We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…

New method improves stability of soft FQI for offline RL.

New method for distributional off-policy evaluation using Bellman residual minimization.

Improved theoretical guarantees for SBEED algorithm.

Selective state-adaptive regularization improves offline RL performance.

A contraction analysis improves model-based RL's error recovery.

FORE evaluates occupancy ratios without requiring Bellman completeness.

In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

This paper introduces a set of algorithms for Monte-Carlo Bayesian reinforcement learning. Firstly, Monte-Carlo estimation of upper bounds on the Bayes-optimal value function is employed to construct an optimistic policy. Secondly, gradient-based algorithms for approximate upper and lower bounds are introduced. Finally…

This paper studies systematic exploration for reinforcement learning with rich observations and function approximation. We introduce a new model called contextual decision processes, that unifies and generalizes most prior settings. Our first contribution is a complexity measure, the Bellman rank, that we show enables …

Second-order estimator improves continuous-time policy evaluation.

New RL algorithm minimizes distributional learning error.

BCRL learns a Bellman complete representation for offline RL policy evaluation.

Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.

This work shows how approximate reward models can significantly improve inference-time scaling.

New approach transfers rewards learned in one environment to reinforcement learning in a new environment.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

A new estimator combines bootstrapping and rollout methods in RL.

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

New algorithms reduce complexity for learning in MDPs with entropy regularization.

Deep nets solve MDPs without high dimensions.

New approach uses PDE learning for faster RL fine-tuning.

Kernel-based methods improve policy evaluation in MRP models.

Finite difference approximations to multi-asset American put option price are considered. The assets are modelled as a multi-dimensional diffusion process with variable drift and volatility. Approximation error of order one quarter with respect to the time discretisation parameter and one half with respect to the space…

We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…

Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.

Study risk-sensitive market making with entropy regularization for better quote control.

The paper explores solutions to the distributional Bellman equation in reinforcement learning.

The paper analyzes risk bounds and Rademacher complexity in batch RL.

KBB algorithm reduces sample complexity for policy evaluation in general state spaces.

Robust Q-learning algorithm resists corrupted rewards.

We obtain the classical Hanner inequalities by the Bellman function method. These inequalities give sharp estimates for the moduli of convexity of Lebesgue spaces. Easy ideas from differential geometry help us to find the Bellman function using neither "magic guesses" nor calculations.

Study shows rate of convergence for particle approximation of PDEs in Wasserstein space.

We consider a new family of operators for reinforcement learning with the goal of alleviating the negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are established, which include showing on a sample path basis that our family of operators preserve optimality an…

Richard Bellman's Principle of Optimality, formulated in 1957, is the heart of dynamic programming, the mathematical discipline which studies the optimal solution of multi-period decision problems. In this paper, we look at the main trading principles of Jesse Livermore, the legendary stock operator whose method was pu…

The paper optimizes portfolios in a financial market with correlated assets using a stochastic volatility model.

Polynomial-time RL algorithm for constant actions under linear Bellman completeness.

New algorithm reduces switching costs in RL beyond linear MDPs.

We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…

One-step Bellman alignment improves online RL by reducing task mismatch.

The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.