The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
A new method calibrates value predictions in offline RL to improve reliability.
problem Difficulty in long-horizon value prediction in offline reinforcement learning.
method Bellman calibration, a weak reliability criterion, and Iterated Bellman Calibration.
result Finite-sample guarantees show that Bellman calibration error is controlled at nonparametric rates.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
Polynomial-time RL algorithm for constant actions under linear Bellman completeness.
problem Efficient online reinforcement learning with few actions.
method Polynomial-time algorithm based on linear function approximation.
result First computationally efficient algorithm for RL with constant actions under linear Bellman completeness.
The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…
New method for off-policy evaluation in POMDPs using future-dependent value functions.
problem Curse of horizon in off-policy evaluation for POMDPs.
method Develops future-dependent value functions and minimax learning method.
result PAC result and Bellman completeness for the proposed OPE estimator.
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximi…
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…
Paper introduces v-CMC linking causality and utility.
problem Linking causality and utility for value theory.
method Developed a new causal independence principle (v-CMC) and proved its equivalence.
result Equivalence of local, global, and decomposition versions of v-CMC.
New method quantifies uncertainty in reinforcement learning models.
problem Quantifying uncertainty over expected cumulative rewards in reinforcement learning.
method Proposes a new uncertainty Bellman equation to more accurately estimate value function variance.
result Our method converges to the true posterior variance over values and improves sample-efficiency.
Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.
problem Improving reinforcement learning by capturing environmental stochasticity and addressing infinite dimensionality.
method Introduces Bellman unbiasedness and proposes SF-LSVI algorithm for provably efficient distributional reinforcement learning.
result Achieves a tight regret bound of O(d_E H^3/2 √K) for distributional reinforcement learning.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
Selective state-adaptive regularization improves offline RL performance.
problem Extrapolation errors and value overestimation in static dataset RL.
method State-adaptive regularization coefficients trust Bellman-driven results selectively.
result Significant improvement in performance on D4RL benchmark.
We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the…
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
A new estimator combines bootstrapping and rollout methods in RL.
problem Combining strengths of bootstrapping and rollout methods in RL.
method Subgraph Bellman operators and fixed point solving.
result Upper bound on error approaches optimal TD variance with additional term.
Deep neural nets approximate high-dimensional HJB equations efficiently.
problem Approximating solutions to high-dimensional HJB equations.
method Deep neural networks for approximating solutions.
result Deep neural networks can approximate solutions without the curse of dimensionality.
Improved machine learning for reservoir optimization problems.
problem Optimizing control in high-dimensional storage problems.
method Modified dynamic programming algorithm with neural networks for Bellman values and conditional cuts.
result Neural networks outperform classical feedforward networks in estimating Bellman values.
We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by O(d) in the action space finite case, and automatically adapts to bias-variance tradeoff. Deep Galerkin Method estimates value function for mean-field control problem.
problem Optimal control of agents with average welfare as the objective.
method Apply DGM to estimate value function and distribution evolution.
result Neural network approximations converge to analytical solution.
One-step Bellman alignment improves online RL by reducing task mismatch.
problem Online RL struggles with task similarity defined by rewards or transitions.
method One-step Bellman alignment and re-weighted targeting (RWT) to correct task mismatch.
result Regret bounds show task shift complexity, not target MDP, affects performance.
A method for calculating multi-portfolio time consistent multivariate risk measures in discrete time is presented. Market models for d assets with transaction costs or illiquidity and possible trading constraints are considered on a finite probability space. The set of capital requirements at each time and state is c…
We characterize the value of swing contracts in continuous time as the unique viscosity solution of a Hamilton-Jacobi-Bellman equation with suitable boundary conditions. The case of contracts with penalties is straightforward, and in that case only a terminal condition is needed. Conversely, the case of contracts with …
We study an optimal execution problem in a continuous-time market model that considers market impact. We formulate the problem as a stochastic control problem and investigate properties of the corresponding value function. We find that right-continuity at the time origin is associated with the strength of market impact…
New approach uses PDE learning for faster RL fine-tuning.
problem Learning optimal control policy for diffusion process.
method Solves variational inequality based on HJB equations.
result Shows fine-tuning can be done via supervised regression.
Trading strategy mimics optimal control with simple heuristic.
problem Maximizing profit in markets with limited optimal control.
method Using the gradient of the Bellman value function as effective microstructure alphas.
result Trading strategy approximates optimal behavior in long-term trades.
The recently proposed distributional approach to reinforcement learning (DiRL) is centered on learning the distribution of the reward-to-go, often referred to as the value distribution. In this work, we show that the distributional Bellman equation, which drives DiRL methods, is equivalent to a generative adversarial n…
In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…
In this paper, we extend the jump-diffusion model proposed by Davis and Lleo to include jumps in asset prices as well as valuation factors. The criterion, following earlier work by Bielecki, Pliska, Nagai and others, is risk-sensitive optimization (equivalent to maximizing the expected growth rate subject to a constrai…
This paper introduces a set of algorithms for Monte-Carlo Bayesian reinforcement learning. Firstly, Monte-Carlo estimation of upper bounds on the Bayes-optimal value function is employed to construct an optimistic policy. Secondly, gradient-based algorithms for approximate upper and lower bounds are introduced. Finally…
KBB algorithm reduces sample complexity for policy evaluation in general state spaces.
problem Policy evaluation in large state spaces with high sample complexity.
method Alternates between fitting Bellman residual and estimating value function via adaptive feature set growth.
result Super-linear convergence rates demonstrated, with reductions in sample complexity.
A new method improves policy evaluation in RL by tracking value uncertainties.
problem Limitations in existing policy evaluation methods for deep RL tasks.
method KOVA (Kalman Optimization for Value Approximation) based on extended Kalman filter.
result KOVA minimizes a regularized objective function that considers parameter and noisy return uncertainties.
Study optimal consumption and investment strategies with leverage constraints using Epstein-Zin utility.
problem Optimal portfolio choice under leverage constraints and Epstein-Zin utility.
method Established viscosity solution to HJB equation, demonstrated smoothness, characterized optimal strategies, derived explicit solutions.
result Explicit solutions for optimal consumption and investment strategies under leverage constraints.
In this paper we propose and analyze a method based on the Riccati transformation for solving the evolutionary Hamilton-Jacobi-Bellman equation arising from the stochastic dynamic optimal allocation problem. We show how the fully nonlinear Hamilton-Jacobi-Bellman equation can be transformed into a quasi-linear paraboli…
Deep learning method proves convergence for high-dimensional PDEs.
problem Solving high-dimensional nonlinear PDEs for mean field control problems.
method Deep Galerkin method (DGM) for Hamilton-Jacobi-Bellman (HJB) equations.
result DGM converges to the true value function of mean field control problems.
Study shows offline RL under Q⋆-approximation and partial coverage is harder than previously thought.
problem Theoretical limits of offline reinforcement learning under Q⋆-approximation and partial coverage. method Introduced a decision-estimation framework to decompose offline RL complexity into decision and value estimation errors.
result Answered the open question by proving sample inefficiency under partial coverage is not guaranteed by Q⋆-realizability and Bellman completeness. Study uses reinforcement learning to optimize portfolios under recursive utility.
problem Improving portfolio allocation using risk-sensitive objectives.
method Approximated certainty equivalent via Monte Carlo, trained actor-critic algorithms (PPO, A2C).
result Recursive-utility agent outperforms discounted baseline in Sharpe ratio, max drawdown, and cumulative return.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
Sequential decision making in the presence of uncertainty and stochastic dynamics gives rise to distributions over state/action trajectories in reinforcement learning (RL) and optimal control problems. This observation has led to a variety of connections between RL and inference in probabilistic graphical models (PGMs)…
The paper proves well-posedness of nonlocal PDEs related to stochastic control problems.
problem Characterizing equilibrium strategies and value functions for time-inconsistent stochastic control problems.
method Method of continuity and Banach's fixed point arguments, with Schauder prior estimates.
result Global well-posedness of nonlocal fully nonlinear PDEs with sharp a-priori estimates.
In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving ve…
The paper develops RL methods for optimal switching between multiple states.
problem Optimal switching between multiple states in continuous time.
method Entropy-regularized exploration, HJB equations, policy improvement, value function convergence.
result The RL algorithm converges to optimal policies as temperature parameter vanishes.
Optimizes portfolio in volatile markets with jumps, providing accurate formulas.
problem Optimizing wealth in a volatile financial market with jumps.
method Analyzes an incomplete stochastic volatility model, derives closed-form portfolio formulas using HJB equation and super-solution/sub-solution.
result Proves accuracy of derived portfolio formulas for both small and finite time horizons.
Study on LOB dynamics using mean-field game theory.
problem Modeling liquidity dynamics in limit order books.
method Mean-field stochastic differential equation and control problem formulation.
result Equilibrium density function of LOB can be derived.
Investigates optimal insurance and reinsurance strategies with incomplete market information.
problem Optimal investment-reinsurance problem for insurance companies with unknown market risk.
method Converted the original problem into a filtered observation problem, applied stochastic control theory, and used Hamilton-Jacobi-Bellman equations.
result Explicit formulas for value function and optimal strategy provided.
Optimizes dividend and reinsurance strategies for correlated insurance lines.
problem Stochastic control of optimal reinsurance and dividend policies for multiple insurance lines.
method Maximizes cumulative discounted dividends using a Hamilton-Jacobi-Bellman equation and finite difference method.
result Provides optimal strategies for transferring risk among reinsurers.