Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid l…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “soft Bellman equation”
New approach transfers rewards learned in one environment to reinforcement learning in a new environment.
problem Transfer of rewards learned using inverse reinforcement learning from one environment to a new, different environment.
method Formulate the problem as a joint system of Bellman equations, develop minimax estimators for the target soft--function, solve the source and target system of equations jointly.
result The coupled approach removes the first-order influence of source Bellman residual error compared to the sequential approach.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Paper introduces RCaI, a risk-sensitive control method using Rényi divergence.
problem Risk-sensitive control in reinforcement learning.
method RCaI extends CaI using Rényi divergence variational inference.
result Risk-sensitive optimal policy can be obtained by solving a soft Bellman equation.
Paper tackles DOCTR-L with SciPhy RL, solving neural PDEs from data.
problem High-dimensional optimal control with stochastic policies.
method Soft HJB equation, Neural PDEs, Physics-Informed Neural Networks.
result Reduces DOCTR-L to solving neural PDEs from data.
The paper analyzes the convergence rates of Q-learning with entropy regularization and linear function approximation.
problem Analyzing the convergence rates of Q-learning with entropy regularization and linear function approximation.
method The paper derives rates of convergence using the high-dimensional central limit theorem, linearization of the soft Bellman recursion, and Gaussian approximation for the leading martingale term.
result The algorithm's last iterate satisfies high-order moment bounds, with a Gaussian approximation bound of order .
This paper analyzes how periodic and soft target updates stabilize linear Q-learning.
problem Theoretical explanation of stabilization mechanisms for linear Q-learning.
method Exact analysis using switched linear system dynamics and the joint spectral radius.
result Periodic and soft target updates can guarantee convergence to the exact projected Q-Bellman solution under specific conditions.
The paper explores solutions to the distributional Bellman equation in reinforcement learning.
problem Distributional reinforcement learning considers complete return distributions, not just expected returns.
method Study existence and uniqueness of solutions to general distributional Bellman equations, linking them to multivariate affine equations.
result Any solution to a distributional Bellman equation can be derived from a multivariate affine distributional equation.
The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcs.LG
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
problem Exponential gap between upper and lower bounds in risk-sensitive RL.
method Identified and addressed deficiencies in existing algorithms and analysis; developed novel analysis and exploration mechanism.
result Improved regret upper bounds over existing ones.
We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…
Study solves optimal portfolio selection using HJB equation.
problem Optimal portfolio selection problem.
method Maximal monotone operator method, Banach fixed-point theorem, Fourier transform, monotone operators technique.
result Existence and uniqueness of solution to HJB equation.
We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm call…
New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.
problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.
method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with -convergence guarantees.
result PAC guarantees for offline RL under partial coverage and realizability conditions.
Deep neural network approximation for high-dimensional parabolic Hamilton-Jacobi-Bellman equationsmath.NA
Deep neural nets approximate high-dimensional HJB equations efficiently.
problem Approximating solutions to high-dimensional HJB equations.
method Deep neural networks for approximating solutions.
result Deep neural networks can approximate solutions without the curse of dimensionality.
Model quantifies uncertainty's impact on European option prices.
problem Uncertainty in market volatility risk affects option pricing.
method Hamilton-Jacobi-Bellman framework and finite element method.
result Dependence of Delta on uncertainty is nonlinear and varied.
Optimizes portfolios with costs, showing existence of optimal strategies.
problem Risk-sensitive portfolio optimization with transaction costs.
method Log-return i.i.d. framework, Bellman equation analysis.
result Existence of optimal strategies for risk-averse and risk-seeking cases.
New RL algorithm minimizes distributional learning error.
problem Improving distributional reinforcement learning for better error minimization.
method Proposes a new model-based algorithm with theoretical minimax optimality.
result Proves minimax optimality for approximating return distributions.
Paper solves investment strategy optimization with deep learning.
problem Maximizing investor utility with optimal asset allocation.
method Solves PDEs with Deep Galerkin method.
result Deep learning algorithm outperforms finite difference method.
The aim of this paper is to construct and analyze solutions to a class of Hamilton-Jacobi-Bellman equations with range bounds on the optimal response variable. Using the Riccati transformation we derive and analyze a fully nonlinear parabolic partial differential equation for the optimal response function. We construct…
We consider a semilinear parabolic degenerated Hamilton-Jacobi-Bellman (HJB) equation with singularity which is related to a stochastic control problem with fuel constraint. The fuel constraint translates into a singular initial condition for the HJB equation. We first propose a transformation based on a change of vari…
In this paper we propose and analyze a method based on the Riccati transformation for solving the evolutionary Hamilton-Jacobi-Bellman equation arising from the stochastic dynamic optimal allocation problem. We show how the fully nonlinear Hamilton-Jacobi-Bellman equation can be transformed into a quasi-linear paraboli…
The paper solves a complex financial optimization problem using a novel mathematical technique.
problem Optimizing portfolio selection in financial markets.
method Maximal monotone operator method and Riccati transformation.
result Existence and uniqueness of a solution to the transformed parabolic equation in a Sobolev space.
Deep Bellman Hedgingq-fin.CP
Deep Bellman Hedging uses reinforcement learning to optimize financial portfolio hedging.
problem Optimizing financial portfolio hedging with derivatives and trading frictions.
method Actor-critic reinforcement learning algorithm with continuous state and action spaces.
result Trained model provides optimal hedge for any initial portfolio and market state.
Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.
problem Analyzing soft-max and policy gradient methods in bandit problems.
method Lyapunov function argument for soft-max and differential equations for policy gradient algorithms.
result Regret bounds for soft-max and a different policy gradient algorithm in bandit problems.
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by in the action space finite case, and automatically adapts to bias-variance tradeoff.
Solves specific mean-field game equations with ODEs.
problem Mean-field game equations in economic applications.
method Reduces coupled PDEs to a quadratically nonlinear system of ODEs.
result Shows specific data leads to solvable ODE system.
Optimal contracts are found for agents with quadratic effort costs.
problem Finding optimal contracts in principal-agent problems with quadratic effort costs.
method Modeling the problem using Hamilton-Jacobi-Bellman (HJB) equations and proving the existence of classical solutions.
result Existence of optimal contracts for agents with quadratic effort costs is proven.
Option Pricing Model for Incomplete Marketcond-mat.stat-mech
The problem of determining the European-style option price in the incomplete market has been examined within the framework of stochastic optimization. An analytic method based on the discrete dynamic programming equation (Bellman equation) has been developed that gives the general formalism for determining the option p…
We solve continuous-time reinforcement learning using distributional Hamilton-Jacobi-Bellman equations.
problem Predicting the distribution of returns in continuous-time, stochastic environments.
method We derive a distributional Hamilton-Jacobi-Bellman equation for Itô diffusions and Feller-Dynkin processes, and propose an algorithm based on a JKO scheme.
result We propose an online control algorithm that can be used to approximately solve the distributional HJB equation.
Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it ta…
New method quantifies uncertainty in reinforcement learning models.
problem Quantifying uncertainty over expected cumulative rewards in reinforcement learning.
method Proposes a new uncertainty Bellman equation to more accurately estimate value function variance.
result Our method converges to the true posterior variance over values and improves sample-efficiency.
New method for off-policy evaluation in POMDPs using future-dependent value functions.
problem Curse of horizon in off-policy evaluation for POMDPs.
method Develops future-dependent value functions and minimax learning method.
result PAC result and Bellman completeness for the proposed OPE estimator.
Deep nets solve MDPs without high dimensions.
problem Solving Bellman equations for MDPs in high dimensions.
method Deep neural networks with ReLU activation approximating payoff and transition functions.
result Deep nets can approximate -functions in polynomially bounded parameters.
Deep learning for HJB PDEs using synthetic data and residual minimization.
problem Solving Hamilton-Jacobi-Bellman PDEs for optimal control problems.
method Gradient-augmented synthetic dataset for supervised learning, residual minimization.
result Improves accuracy and efficiency of deep learning for HJB PDEs.
We study the structure of a simple dynamic optimization problem consisting of one state and one control variable, from a physicist's point of view. By using an analogy to a physical model, we study this system in the classical and quantum frameworks. Classically, the dynamic optimization problem is equivalent to a clas…
New method uses TT approximations to solve HJB equations for efficient sampling.
problem Efficiently sampling from complex probability densities.
method Direct time integration of HJB equations using Tensor Train compression.
result Sample-free, dimensionality-avoiding integration method.
We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the…
Optimizes control of infectious disease spread using stochastic methods.
problem Optimizing control of highly infectious diseases like COVID-19.
method Reformulated Hamilton-Jacobi-Bellman equation as stochastic minimum principle, leading to forward-backward stochastic differential equations.
result Numerous numerical solutions presented under various scenarios.
We consider a general class of non-linear Bellman equations. These open up a design space of algorithms that have interesting properties, which has two potential advantages. First, we can perhaps better model natural phenomena. For instance, hyperbolic discounting has been proposed as a mathematical model that matches …
Optimal reinsurance strategies for multi-line insurance companies.
problem Choosing the best dynamic reinsurance policies for multi-line insurance companies.
method Characterized the optimal survival function as the unique nondecreasing viscosity solution of the HJB equation, solved numerically using the finite difference method.
result Provided proof of convergence of numerical solution to the survival probability function.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
Stochastic differential equation approximation for linear TD(0) under Markovian noise
problem Temporal-difference learning with linear function approximation
method Stochastic differential equation approximation
result Explains the constant-stepsize error floor
This paper surveys some recent results on existence, uniqueness and removable singularities for fully nonlinear differential equations on manifolds. The discussion also treats restriction theorems and the strong Bellman principle.
New RL formulation for maximizing maximum reward in molecule generation.
problem Traditional RL frameworks do not fit real-world applications like drug discovery.
method Formulated a new objective function to maximize maximum reward, derived Bellman equation, introduced operators, and proved convergence.
result Achieved state-of-the-art results in molecule generation.
In this paper, we extend the jump-diffusion model proposed by Davis and Lleo to include jumps in asset prices as well as valuation factors. The criterion, following earlier work by Bielecki, Pliska, Nagai and others, is risk-sensitive optimization (equivalent to maximizing the expected growth rate subject to a constrai…
A new option pricing model handles non-constant risk aversion and transaction costs.
problem Deriving a pricing model for options with varying risk aversion.
method Developed a transformation method to solve the penalized nonlinear PDE and used finite difference discretization.
result Derived bounds on option prices and proposed a numerical scheme.
Machine Learning and Hamilton-Jacobi-Bellman Equation for Optimal Decumulation: a Comparison Studymath.OC
A neural network approach solves optimal decumulation problems for pension plans.
problem Optimal asset allocation and withdrawal strategies for DC pension holders.
method Data-driven neural network optimization with customized activation functions.
result The neural network approach learns near-optimal solutions comparable to HJB PDE methods.