The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
The paper shows how to learn near-optimal behavior in reinforcement learning with rich observations.
problem Learning near-optimal behavior in reinforcement learning with rich observations and function approximation.
method Introduces a new model called contextual decision processes and a new algorithm that engages in systematic exploration to learn these processes with low Bellman rank.
result The algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters.
New Bellman error estimator improves offline model selection performance.
problem Selecting the best policy from logged data using mean squared Bellman error.
method Developed a more accurate estimator of MSBE and analyzed conditions for successful OMS.
result New estimator achieves impressive offline model selection performance on diverse tasks.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
A new method calibrates value predictions in offline RL to improve reliability.
problem Difficulty in long-horizon value prediction in offline reinforcement learning.
method Bellman calibration, a weak reliability criterion, and Iterated Bellman Calibration.
result Finite-sample guarantees show that Bellman calibration error is controlled at nonparametric rates.
New RL algorithm minimizes distributional learning error.
problem Improving distributional reinforcement learning for better error minimization.
method Proposes a new model-based algorithm with theoretical minimax optimality.
result Proves minimax optimality for approximating return distributions.
The paper analyzes risk bounds and Rademacher complexity in batch RL.
problem Estimating/minimizing Bellman error with general value function approximation.
method Characterizes generalization performance using Rademacher complexities of function classes.
result Risk bounds and Rademacher complexities provide insights into batch RL.
Two new algorithms improve Q* approximation in batch RL with linear error propagation.
problem Improving Q* approximation in batch reinforcement learning.
method Two novel algorithms that estimate Bellman error directly, without quadratic dependence.
result Linear-in-horizon error propagation for batch RL algorithms.
Deep learning for HJB PDEs using synthetic data and residual minimization.
problem Solving Hamilton-Jacobi-Bellman PDEs for optimal control problems.
method Gradient-augmented synthetic dataset for supervised learning, residual minimization.
result Improves accuracy and efficiency of deep learning for HJB PDEs.
New methods for off-policy evaluation in reinforcement learning.
problem Improving off-policy evaluation in reinforcement learning.
method Developed MWL and MQL estimators for importance weights and value functions.
result MWL and MQL provide a unified view of algorithms in reinforcement learning.
We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Improved theoretical guarantees for SBEED algorithm.
problem Theoretical analysis of SBEED algorithm's performance.
method Near-optimal performance guarantee based on function classes and distribution shift.
result Improved guarantees for SBEED in terms of horizon and sample size.
We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…
Selective state-adaptive regularization improves offline RL performance.
problem Extrapolation errors and value overestimation in static dataset RL.
method State-adaptive regularization coefficients trust Bellman-driven results selectively.
result Significant improvement in performance on D4RL benchmark.
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximi…
Paper analyzes AVI scheme for noisy Bellman approximations.
problem Analyzing stability and convergence of noisy value iteration.
method Uses neural networks to approximate Bellman operator, considers biased approximations and sampling errors.
result Verifiable conditions for stability and convergence of AVI.
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
Softmax Bellman operator improves Q-function performance in RL despite sub-optimality.
problem Softmax Bellman operator's impact on value functions in RL is problematic.
method Revisited theoretical properties of softmax Bellman operator, proving convergence and overestimation reduction.
result Softmax Bellman operator leads to superior policies in practice, even outperforming double Q-learning.
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
This paper introduces a set of algorithms for Monte-Carlo Bayesian reinforcement learning. Firstly, Monte-Carlo estimation of upper bounds on the Bayes-optimal value function is employed to construct an optimistic policy. Secondly, gradient-based algorithms for approximate upper and lower bounds are introduced. Finally…
Second-order estimator improves continuous-time policy evaluation.
problem Estimating value surfaces from discrete data with time-inhomogeneous dynamics.
method Moment-matching coefficients for high-order generator regression.
result Second-order estimator consistently outperforms Bellman baseline.
New algorithm reduces switching costs in RL beyond linear MDPs.
problem Costly policy switching in reinforcement learning.
method ELEANOR-LowSwitching algorithm for linear Bellman-complete MDPs.
result Achieves near-optimal regret with logarithmic switching cost.
Optimal reinsurance minimizes expected discounted penalty in a Cramer-Lundberg model.
problem Minimizing expected discounted penalty functions in a Cramer-Lundberg model.
method Using optimal stochastic control theory and solving the Hamilton-Jacobi-Bellman equation.
result Existence and uniqueness of the solution found by the method.
This paper explores how IV methods can improve Q-function estimates in offline policy evaluation.
problem Confounding in estimating Q-function using reinforcement learning.
method Integrates IV techniques into offline policy evaluation (OPE) to improve Q-function estimates.
result State-of-the-art OPE methods are closely matched in performance by some IV methods.
BCRL learns a Bellman complete representation for offline RL policy evaluation.
problem Learning a Q-function efficiently from offline data.
method BCRL learns a linear Bellman complete representation directly from data, enabling efficient OPE.
result BCRL achieves competitive OPE error and outperforms FQE in certain scenarios.
A new method improves policy evaluation in RL by tracking value uncertainties.
problem Limitations in existing policy evaluation methods for deep RL tasks.
method KOVA (Kalman Optimization for Value Approximation) based on extended Kalman filter.
result KOVA minimizes a regularized objective function that considers parameter and noisy return uncertainties.
Optimizes portfolios with costs, showing existence of optimal strategies.
problem Risk-sensitive portfolio optimization with transaction costs.
method Log-return i.i.d. framework, Bellman equation analysis.
result Existence of optimal strategies for risk-averse and risk-seeking cases.
Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.
problem Challenges in offline reinforcement learning with deep neural networks and data dependence.
method Establishes estimation error for pessimistic offline RL using neural network approximation with C \mathcal{C} C -mixing data. result Explicit efficiency of deep adversarial offline RL frameworks demonstrated with two converging error components.
This work shows how approximate reward models can significantly improve inference-time scaling.
problem Improving the efficiency of inference for large language models.
method Identifying the Bellman error of approximate reward models and using Sequential Monte Carlo (SMC) for inference.
result Approximate reward models can reduce computational complexity from exponential to polynomial in T T T . New approach transfers rewards learned in one environment to reinforcement learning in a new environment.
problem Transfer of rewards learned using inverse reinforcement learning from one environment to a new, different environment.
method Formulate the problem as a joint system of Bellman equations, develop minimax estimators for the target soft- q q q -function, solve the source and target system of equations jointly. result The coupled approach removes the first-order influence of source Bellman residual error compared to the sequential approach.
Stochastic differential equation approximation for linear TD(0) under Markovian noise
problem Temporal-difference learning with linear function approximation
method Stochastic differential equation approximation
result Explains the constant-stepsize error floor
New reinforcement learning operators improve performance and robustness.
problem Improving reinforcement learning algorithms to handle approximation errors.
method Developed a new family of robust stochastic operators.
result Preserves optimality and increases action gap on sample paths.
KOVA optimizes value functions using Kalman filtering, improving parameter uncertainty.
problem Improving parameter uncertainty in value function approximation.
method KOVA uses a trust region approach with a Bayesian perspective and Kalman filtering.
result KOVA provides more reliable parameter estimates and value function approximations.
New algorithm BEAR reduces instability in off-policy Q-learning.
problem High sensitivity of off-policy Q-learning methods to data distribution.
method Identified and mitigated bootstrapping error through constrained action selection.
result BEAR algorithm learns robustly from various off-policy distributions.
Proposes a new reinforcement learning algorithm using Q-function.
problem Optimal control in Markov Decision Processes (MDPs).
method Regularized linear-programming formulation, Q-function, saddle-point optimization.
result Demonstrates effectiveness on various benchmark problems.
A new estimator combines bootstrapping and rollout methods in RL.
problem Combining strengths of bootstrapping and rollout methods in RL.
method Subgraph Bellman operators and fixed point solving.
result Upper bound on error approaches optimal TD variance with additional term.
Unified coverage analysis for linear off-policy evaluation in reinforcement learning.
problem Lack of a unified understanding of coverage parameters in linear off-policy evaluation.
method Developed a novel finite-sample analysis for LSTDQ algorithm, introducing feature-dynamics coverage.
result Unified understanding of coverage parameters in linear off-policy evaluation.
Proposes RFQI for robust RL using offline data.
problem Learning robust policies in the presence of model uncertainty.
method RFQI algorithm using offline data to learn optimal robust policy.
result RFQI learns near-optimal robust policy under standard assumptions.
Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a γ \sqrtγ γ -contraction modulus (where γ γ γ is the …
New algorithms reduce complexity for learning in MDPs with entropy regularization.
problem Efficient learning for MDPs with large or continuous state and action spaces.
method Multilevel Monte Carlo (MLMC) algorithms integrating fixed-point iteration and stochastic approximation of the Bellman operator.
result MLMC with unbiased approximation of the Bellman operator achieves polynomial sample complexity.
Paper introduces a new framework to improve sample efficiency in POMDPs learning.
problem Challenges in off-policy evaluation for POMDPs, especially with hidden states.
method Exploits the metric structure of belief space to relax coverage assumptions.
result Unified analysis technique yields tighter error bounds and sample efficiency improvements.
Deep nets solve MDPs without high dimensions.
problem Solving Bellman equations for MDPs in high dimensions.
method Deep neural networks with ReLU activation approximating payoff and transition functions.
result Deep nets can approximate Q Q Q -functions in polynomially bounded parameters. New approach uses PDE learning for faster RL fine-tuning.
problem Learning optimal control policy for diffusion process.
method Solves variational inequality based on HJB equations.
result Shows fine-tuning can be done via supervised regression.
This work addresses time inconsistency in risk measures and develops a dynamic programming principle for risk minimization problems.
problem Time inconsistency in optimized certainty equivalents (OCEs) risk measures.
method Enlargement of state space to achieve a substitute for time consistency, derivation of dynamic programming principle.
result Characterization of the value function via viscosity solutions of Hamilton--Jacobi--Bellman--Issacs equations.
Kernel-based methods improve policy evaluation in MRP models.
problem Estimating value functions in infinite-horizon discounted MRP models.
method Kernel-based temporal difference methods using reproducing kernel Hilbert spaces.
result Optimal error bounds derived for the kernel-based LSTD estimate.