arXiv research

Paper studies offline RL with linear approx, focusing on inherent Bellman error.

New method stabilizes FQE by reweighting Bellman targets.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

New Bellman error estimator improves offline model selection performance.

A new method calibrates value predictions in offline RL to improve reliability.

New algorithm reduces switching costs in RL beyond linear MDPs.

New method learns policies from offline data using operator models.

The problem of determining the European-style option price in the incomplete market has been examined within the framework of stochastic optimization. An analytic method based on the discrete dynamic programming equation (Bellman equation) has been developed that gives the general formalism for determining the option p…

Two new algorithms improve Q* approximation in batch RL with linear error propagation.

DSPI connects natural policy gradient to policy iteration, proving global convergence.

We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…

New method improves stability of soft FQI for offline RL.

New method for distributional off-policy evaluation using Bellman residual minimization.

Improved theoretical guarantees for SBEED algorithm.

Selective state-adaptive regularization improves offline RL performance.

A contraction analysis improves model-based RL's error recovery.

FORE evaluates occupancy ratios without requiring Bellman completeness.

In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

This paper introduces a set of algorithms for Monte-Carlo Bayesian reinforcement learning. Firstly, Monte-Carlo estimation of upper bounds on the Bayes-optimal value function is employed to construct an optimistic policy. Secondly, gradient-based algorithms for approximate upper and lower bounds are introduced. Finally…

This paper studies systematic exploration for reinforcement learning with rich observations and function approximation. We introduce a new model called contextual decision processes, that unifies and generalizes most prior settings. Our first contribution is a complexity measure, the Bellman rank, that we show enables …

Second-order estimator improves continuous-time policy evaluation.

New RL algorithm minimizes distributional learning error.

BCRL learns a Bellman complete representation for offline RL policy evaluation.

Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.

This work shows how approximate reward models can significantly improve inference-time scaling.

New approach transfers rewards learned in one environment to reinforcement learning in a new environment.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

New algorithm learns optimal policies without explicit rewards.

A new estimator combines bootstrapping and rollout methods in RL.

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

New algorithms reduce complexity for learning in MDPs with entropy regularization.

New method reduces sample complexity for robust reinforcement learning.

Deep nets solve MDPs without high dimensions.

New approach uses PDE learning for faster RL fine-tuning.

Kernel-based methods improve policy evaluation in MRP models.

Faster algorithms for solving multichain MDPs under average-reward criterion.

Finite difference approximations to multi-asset American put option price are considered. The assets are modelled as a multi-dimensional diffusion process with variable drift and volatility. Approximation error of order one quarter with respect to the time discretisation parameter and one half with respect to the space…

We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…

This paper identifies and analyzes biases in risk-adjusted index weighting methods, affecting social welfare and market fairness.

Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.

We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is obs…

Study risk-sensitive market making with entropy regularization for better quote control.

The paper explores solutions to the distributional Bellman equation in reinforcement learning.

The paper analyzes risk bounds and Rademacher complexity in batch RL.

KBB algorithm reduces sample complexity for policy evaluation in general state spaces.

Robust Q-learning algorithm resists corrupted rewards.

We describe a new approach for managing aleatoric uncertainty in the Reinforcement Learning (RL) paradigm. Instead of selecting actions according to a single statistic, we propose a distributional method based on the second-order stochastic dominance (SSD) relation. This compares the inherent dispersion of random retur…