arXiv research

The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

DSPI connects natural policy gradient to policy iteration, proving global convergence.

In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…

Study solves optimal portfolio selection using HJB equation.

New algorithms reduce complexity for learning in MDPs with entropy regularization.

Richard Bellman's Principle of Optimality, formulated in 1957, is the heart of dynamic programming, the mathematical discipline which studies the optimal solution of multi-period decision problems. In this paper, we look at the main trading principles of Jesse Livermore, the legendary stock operator whose method was pu…

The paper tackles non-cumulative objectives in reinforcement learning and proposes modifications to existing algorithms.

New method improves stability of soft FQI for offline RL.

New method stabilizes FQE by reweighting Bellman targets.

A new estimator combines bootstrapping and rollout methods in RL.

One-step Bellman alignment improves online RL by reducing task mismatch.

We consider a new family of operators for reinforcement learning with the goal of alleviating the negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are established, which include showing on a sample path basis that our family of operators preserve optimality an…

The paper solves a complex financial optimization problem using a novel mathematical technique.

New method learns policies from offline data using operator models.

New RL formulation for maximizing maximum reward in molecule generation.

FORE evaluates occupancy ratios without requiring Bellman completeness.

We consider a general class of non-linear Bellman equations. These open up a design space of algorithms that have interesting properties, which has two potential advantages. First, we can perhaps better model natural phenomena. For instance, hyperbolic discounting has been proposed as a mathematical model that matches …

The paper tackles data-driven optimal control of unknown nonlinear systems using RKHS.

Market makers optimize trading with a new implicit scheme for complex inequalities.

A framework solves parametric families of MFGs efficiently.

Second-order estimator improves continuous-time policy evaluation.

Stochastic differential equation approximation for linear TD(0) under Markovian noise

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the …

Paper establishes DRL for high-dimensional rewards.

Novel framework proves fast RL convergence in continuous spaces.

Survey of mathematical foundations for reinforcement learning.

New algorithms solve robust MDPs efficiently, significantly faster than existing methods.

This work is motivated by numerical solutions to Hamilton-Jacobi-Bellman quasi-variational inequalities (HJBQVIs) associated with combined stochastic and impulse control problems. In particular, we consider (i) direct control, (ii) penalized, and (iii) semi-Lagrangian discretization schemes applied to the HJBQVI proble…

This paper investigates sufficient conditions for a Feynman-Kac functional up to an exit time to be the generalized viscosity solution of a Dirichlet problem. The key ingredient is to find out the continuity of exit operator under Skorokhod topology, which reveals the intrinsic connection between overfitting Dirichlet …

Unified theory of $θ$-expectations derived from chaotic dynamics.

The paper explores solutions to the distributional Bellman equation in reinforcement learning.

Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variant of Bellman operator that is not necessarily a contraction. As a result, they may easily …

Kernel-based methods improve policy evaluation in MRP models.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…

Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.

We consider a stochastic control problem with the assumption that the system is controlled until the state process breaks the fixed barrier. Assuming some general conditions, it is proved that the resulting Hamilton Jacobi Bellman equations has smooth solution. The aforementioned result is used to solve the optimal div…

Core-Halo solves large-scale fixed-point problems by decentralizing updates.

We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm call…

Polynomial-time RL algorithm for constant actions under linear Bellman completeness.

A new method calibrates value predictions in offline RL to improve reliability.

New method reduces sample complexity for robust reinforcement learning.

Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.

New Bellman error estimator improves offline model selection performance.

Study risk-sensitive market making with entropy regularization for better quote control.

The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.

We consider Markov Decision Problems defined over continuous state and action spaces, where an autonomous agent seeks to learn a map from its states to actions so as to maximize its long-term discounted accumulation of rewards. We address this problem by considering Bellman's optimality equation defined over action-val…