The paper shows how to learn near-optimal behavior in reinforcement learning with rich observations.
problem Learning near-optimal behavior in reinforcement learning with rich observations and function approximation.
method Introduces a new model called contextual decision processes and a new algorithm that engages in systematic exploration to learn these processes with low Bellman rank.
result The algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters.
Paper tackles online learning in large MDPs with low Bellman rank using AVE algorithm.
problem Online learning of MDPs with large state spaces.
method Develops AVE algorithm inspired by OLIVE, using contextual bandit problems and elimination steps.
result Achieves n \sqrt{n} n -regret for learning optimal value function in MDPs with function approximation and low Bellman rank. Survey of reinforcement learning guarantees with data constraints.
problem Guaranteeing near-optimal policies with limited data in reinforcement learning.
method Coverage-Structure-Objective (CSO) framework to decompose sample complexity results.
result Progress on PAC guarantees for reinforcement learning, covering various models and settings.
New BE dimension measure reveals rich RL problems with sample-efficient algorithms.
problem Finding sample-efficient algorithms for complex RL problems.
method Introducing Bellman Eluder (BE) dimension and designing GOLF and OLIVE algorithms.
result GOLF and OLIVE algorithms learn near-optimal policies for low BE dimension problems with polynomial samples.
Bayesian framework for optimal sampling and selection in ranking problems.
problem Optimal sampling and selection in statistical ranking and selection.
method Formulated as a stochastic control problem, derived Bellman equation, value function approximation for optimal policy.
result Approximately optimal allocation policy with one-step-ahead and asymptotic optimality for independent normal distributions.
Efficiently samples complex distributions using tensor train format.
problem Sampling from high-dimensional complex probability densities efficiently.
method Integrates tensor train format with backward stochastic differential equations (BSDEs) for fast, robust, and accurate sampling.
result Improved efficiency in sampling from challenging target distributions.
New method uses TT approximations to solve HJB equations for efficient sampling.
problem Efficiently sampling from complex probability densities.
method Direct time integration of HJB equations using Tensor Train compression.
result Sample-free, dimensionality-avoiding integration method.
The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
New objective function preserves Bellman's principle for policy gradient.
problem Lack of objective function capturing Bellman's principle optimally.
method Proposed a new objective function and its gradient.
result Preserves Bellman's principle of optimality in policy gradient.
A new method calibrates value predictions in offline RL to improve reliability.
problem Difficulty in long-horizon value prediction in offline reinforcement learning.
method Bellman calibration, a weak reliability criterion, and Iterated Bellman Calibration.
result Finite-sample guarantees show that Bellman calibration error is controlled at nonparametric rates.
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
problem Exponential gap between upper and lower bounds in risk-sensitive RL.
method Identified and addressed deficiencies in existing algorithms and analysis; developed novel analysis and exploration mechanism.
result Improved regret upper bounds over existing ones.
New Bellman error estimator improves offline model selection performance.
problem Selecting the best policy from logged data using mean squared Bellman error.
method Developed a more accurate estimator of MSBE and analyzed conditions for successful OMS.
result New estimator achieves impressive offline model selection performance on diverse tasks.
The paper explores solutions to the distributional Bellman equation in reinforcement learning.
problem Distributional reinforcement learning considers complete return distributions, not just expected returns.
method Study existence and uniqueness of solutions to general distributional Bellman equations, linking them to multivariate affine equations.
result Any solution to a distributional Bellman equation can be derived from a multivariate affine distributional equation.
Softmax Bellman operator improves Q-function performance in RL despite sub-optimality.
problem Softmax Bellman operator's impact on value functions in RL is problematic.
method Revisited theoretical properties of softmax Bellman operator, proving convergence and overestimation reduction.
result Softmax Bellman operator leads to superior policies in practice, even outperforming double Q-learning.
We obtain the classical Hanner inequalities by the Bellman function method. These inequalities give sharp estimates for the moduli of convexity of Lebesgue spaces. Easy ideas from differential geometry help us to find the Bellman function using neither "magic guesses" nor calculations.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
Study shows offline RL under Q ⋆ Q^\star Q ⋆ -approximation and partial coverage is harder than previously thought.
problem Theoretical limits of offline reinforcement learning under Q ⋆ Q^\star Q ⋆ -approximation and partial coverage. method Introduced a decision-estimation framework to decompose offline RL complexity into decision and value estimation errors.
result Answered the open question by proving sample inefficiency under partial coverage is not guaranteed by Q ⋆ Q^\star Q ⋆ -realizability and Bellman completeness. Richard Bellman's Principle of Optimality, formulated in 1957, is the heart of dynamic programming, the mathematical discipline which studies the optimal solution of multi-period decision problems. In this paper, we look at the main trading principles of Jesse Livermore, the legendary stock operator whose method was pu…
Polynomial-time RL algorithm for constant actions under linear Bellman completeness.
problem Efficient online reinforcement learning with few actions.
method Polynomial-time algorithm based on linear function approximation.
result First computationally efficient algorithm for RL with constant actions under linear Bellman completeness.
New RL algorithms show model-based methods are more efficient than model-free ones in complex decision processes.
problem Efficient reinforcement learning in contextual decision processes with strategic exploration.
method Design of new model-based RL algorithms with sample complexity governed by witness rank.
result Exponential separation between model-based and model-free RL in some rich-observation settings.
New insights show coverage conditions are crucial for efficient online reinforcement learning.
problem The role of coverage conditions in determining sample complexity of offline reinforcement learning.
method Established a connection between coverage conditions and sample efficiency in online reinforcement learning.
result Coverability, a structural property of MDPs, enables sample-efficient exploration in online reinforcement learning.
BCRL learns a Bellman complete representation for offline RL policy evaluation.
problem Learning a Q-function efficiently from offline data.
method BCRL learns a linear Bellman complete representation directly from data, enabling efficient OPE.
result BCRL achieves competitive OPE error and outperforms FQE in certain scenarios.
We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…
One-step Bellman alignment improves online RL by reducing task mismatch.
problem Online RL struggles with task similarity defined by rewards or transitions.
method One-step Bellman alignment and re-weighted targeting (RWT) to correct task mismatch.
result Regret bounds show task shift complexity, not target MDP, affects performance.
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by O ( d ) \mathcal{O}(d) O ( d ) in the action space finite case, and automatically adapts to bias-variance tradeoff. Paper introduces dynamic strategies for multi-period investment models.
problem Optimizing investment strategies over multiple periods with risk and return considerations.
method Developed a Bellman principle for discrete time multi-period mean-variance models, leading to dynamic optimal strategies and efficient frontiers.
result Dynamic optimal strategies can achieve higher returns with lower risk compared to the 1/n strategy.
This paper extends the Bellman equation for reinforcement learning to explore uncertainty.
problem Exploration in reinforcement learning, focusing on uncertainty.
method Uncertainty Bellman Equation (UBE) to connect uncertainty at any time-step to expected uncertainties at subsequent time-steps.
result UBE's unique fixed point provides an upper bound on Q-values variance, improving DQN performance.
Model quantifies uncertainty's impact on European option prices.
problem Uncertainty in market volatility risk affects option pricing.
method Hamilton-Jacobi-Bellman framework and finite element method.
result Dependence of Delta on uncertainty is nonlinear and varied.
Study solves optimal portfolio selection using HJB equation.
problem Optimal portfolio selection problem.
method Maximal monotone operator method, Banach fixed-point theorem, Fourier transform, monotone operators technique.
result Existence and uniqueness of solution to HJB equation.
Paper analyzes AVI scheme for noisy Bellman approximations.
problem Analyzing stability and convergence of noisy value iteration.
method Uses neural networks to approximate Bellman operator, considers biased approximations and sampling errors.
result Verifiable conditions for stability and convergence of AVI.
Deep Bellman Hedging uses reinforcement learning to optimize financial portfolio hedging.
problem Optimizing financial portfolio hedging with derivatives and trading frictions.
method Actor-critic reinforcement learning algorithm with continuous state and action spaces.
result Trained model provides optimal hedge for any initial portfolio and market state.
The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.
problem High variance in off-policy TD-learning due to importance sampling.
method Derives finite-sample bounds for off-policy TD-like algorithms using generalized Bellman operators.
result First-known finite-sample guarantees for several off-policy TD algorithms.
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
New non-linear models for better preference predictions.
problem Improving models for preference orderings.
method Developed a general class of non-linear Bellman equations.
result New models can make different predictions under various circumstances.
Deep neural nets approximate high-dimensional HJB equations efficiently.
problem Approximating solutions to high-dimensional HJB equations.
method Deep neural networks for approximating solutions.
result Deep neural networks can approximate solutions without the curse of dimensionality.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Optimizes portfolios with costs, showing existence of optimal strategies.
problem Risk-sensitive portfolio optimization with transaction costs.
method Log-return i.i.d. framework, Bellman equation analysis.
result Existence of optimal strategies for risk-averse and risk-seeking cases.
New RL algorithm minimizes distributional learning error.
problem Improving distributional reinforcement learning for better error minimization.
method Proposes a new model-based algorithm with theoretical minimax optimality.
result Proves minimax optimality for approximating return distributions.
Market makers optimize trading with a new implicit scheme for complex inequalities.
problem Optimizing trading in a limit order book with stochastic and impulse control.
method Implicit numerical scheme coupled with policy iteration algorithm.
result Convergence to the unique viscosity solution of the HJBQVI.
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximi…
Solves complex equation with singularities using transformations and numerical methods.
problem Solving a semilinear parabolic HJB equation with a singular initial condition.
method Transformed the equation to remove singularity, then constructed numerical schemes.
result Proved convergence of numerical schemes for the transformed equation.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
Deep learning for HJB PDEs using synthetic data and residual minimization.
problem Solving Hamilton-Jacobi-Bellman PDEs for optimal control problems.
method Gradient-augmented synthetic dataset for supervised learning, residual minimization.
result Improves accuracy and efficiency of deep learning for HJB PDEs.
New method solves continuous time mean-variance model for consistent investment strategy.
problem Time-consistent optimal strategy for continuous time mean-variance model.
method Developed a new Bellman principle method.
result Obtained a time-consistent dynamic optimal strategy.
Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.
problem Improving reinforcement learning by capturing environmental stochasticity and addressing infinite dimensionality.
method Introduces Bellman unbiasedness and proposes SF-LSVI algorithm for provably efficient distributional reinforcement learning.
result Achieves a tight regret bound of O(d_E H^3/2 √K) for distributional reinforcement learning.
A new estimator combines bootstrapping and rollout methods in RL.
problem Combining strengths of bootstrapping and rollout methods in RL.
method Subgraph Bellman operators and fixed point solving.
result Upper bound on error approaches optimal TD variance with additional term.