Proposes SOR Q-learning for faster optimal value function computation in RL.
problem Finding optimal value function in Markov Decision Processes (MDPs).
method Successive Over-Relaxation (SOR) applied to Q-learning algorithm.
result SOR Q-learning converges faster to optimal value function compared to standard Q-learning.
Geometric approach improves reinforcement learning representation.
problem Improving reinforcement learning representation learning.
method Formal evidence through geometric properties of value functions.
result Optimizing value functions reduces to predicting adversarial value functions (AVFs).
POLO framework enables efficient learning and exploration in model-based control.
problem Efficient learning and exploration in model-based control settings.
method Combines local model-based control, global value function learning, and exploration.
result POLO framework accelerates value function learning and enables better policies.
This paper interpolates reward functions to predict optimal value functions in MORL.
problem Finding optimal value functions in MORL requires recomputing for each set of weights.
method Interpolating reward function weights to smooth value function transformations.
result Smooth interpolation of optimal value functions over reward function weights.
A new method separates long-term value functions into components for better reinforcement learning.
problem Learning long-term goals in reinforcement learning settings with temporal discounting.
method TD(Δ) learning, which breaks down value functions into components based on discount factors. result TD(Δ) learning improves scalability and performance over standard TD learning in certain settings. DMPC combines MPC and value function estimation for efficient control tasks.
problem Efficiently solve control tasks with sparse and binary reward signals.
method Actor-critic algorithm combining MPC and value function estimation.
result DMPC actor minimizes an upper bound of cross-entropy to optimal policy.
We analyze the Gambler's problem, finding a complex, fractal value function.
problem Simple reinforcement learning problem where a gambler tries to reach a target.
method Exact formula derivation for discrete and continuous cases.
result The optimal value function is fractal, self-similar, and has high-frequency components.
Paper introduces a new value function for state transitions and optimal policy learning.
problem Learning optimal policies from state transitions and actions.
method Develops a forward dynamics model to maximize a novel value function Q(s,s′). result Demonstrates benefits in value function transfer, redundant action spaces, and off-policy learning.
Develops a new method for statistical optimal allocation problems.
problem Statistical optimal allocation problems with constraints.
method Functional differentiability approach and Hadamard differentiability of value functions.
result Validates margin assumption for fast convergence rate of plug-in methods.
We study an optimal execution problem with uncertain market impact to derive a more realistic market model. We construct a discrete-time model as a value function for optimal execution. Market impact is formulated as the product of a deterministic part increasing with execution volume and a positive stochastic noise pa…
This paper considers a utility maximization and optimal asset allocation problem in the presence of a stochastic endowment that cannot be fully hedged through trading in the financial market. After studying continuity properties of the value function for general utility functions, we rely on the dynamic programming app…
Method explains MDP optimal policies and reward collection.
problem Understanding optimal policies and reward collection in MDPs.
method Deterministic MDPs with a method to map state space and explain rewards.
result Algorithm determines which rewards will be collected and their frequency.
PPG separates policy and value function training phases for better reinforcement learning efficiency.
problem Challenges in traditional reinforcement learning methods for policy and value function optimization.
method Integrates Phasic Policy Gradient framework that splits policy and value function training into distinct phases.
result Significantly improves sample efficiency on Procgen Benchmark compared to PPO.
Optimizes dividend payout in insurance wealth process with stochastic interest rate.
problem Maximizing expected discounted dividends up to ruin in insurance wealth process.
method Modelled compound Poisson process with stochastic interest rate, solved using HJB equation.
result Explicit expression for value function and optimal strategy in geometric Brownian motion case.
Deep RBVFs improve continuous control in RL.
problem Challenges in finding optimal actions for continuous actions in RL.
method Introduced deep radial-basis value functions (RBVFs) for continuous control.
result RBF-DQN significantly outperforms value-function-only baselines and is competitive with actor-critic algorithms.
Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function V. Two novel algorithms are proposed to approximate the true value…
We study the regularity properties of the value function associated with an affine optimal control problem with quadratic cost plus a potential, for a fixed final time and initial point. Without assuming any condition on singular minimizers, we prove that the value function is continuous on an open and dense subset of …
The paper examines smoothness of value function in consumption-investment models with borrowing constraints.
problem Investor's optimal consumption and investment under consumption-wealth utility and borrowing constraint.
method Second-order smoothness of value function, optimal consumption-investment policy in feedback form, smooth fit condition.
result The value function is second-order smooth and the constraint is binding under certain conditions.
Boosts Q-learning by using value function bounds.
problem Efficiently solving new tasks using past experience.
method Derives double-sided bounds on optimal value function and uses them to update Q-function.
result Boosted training performance through alternative Q-function update method.
TensorPlan algorithm finds δ-optimal policies with poly(H,d) queries under linearly realizable state-value function.
problem Efficient planning in MDPs with linearly realizable state-value function.
method TensorPlan algorithm using poly((dH/δ)A) simulator queries. result First algorithm with polynomial query complexity using only linear-realizability of a single competing value function.
Closed-form approximations for multi-asset market making models.
problem Challenges in numerical approximation for large multi-asset models.
method Proposed closed-form approximations for value functions and optimal quotes.
result New closed-form approximations for optimal quotes in multi-asset markets.
Addressing RL's agent-environment boundary issues, a novel analysis ensures optimal value functions are invariant.
problem Fundamental RL concepts like value functions are not uniquely defined due to the agent-environment boundary.
method A boundary-invariant analysis of Fitted Q-Iteration, ensuring optimality guarantees are independent of the boundary choice.
result Theoretical analyses of RL algorithms, including Fitted Q-Iteration, are made invariant to the boundary choice.
New concept of Blackwell regret for reinforcement learning with sparse rewards.
problem Sparse rewards in long horizon MDPs.
method Formalization of myopic discount factors, value functions, and policies in terms of Blackwell optimality; introduction of Blackwell regret.
result Selecting a discount factor for zero Blackwell regret becomes arbitrarily hard in long horizon MDPs.
We study regularity properties of the dynamic value functions of primal and dual problems of optimal investing for utility functions defined on the whole real line. Relations between decomposition terms of value processes of primal and dual problems and between optimal solutions of basic and conditional utility maximiz…
The Lax-Hopf formula simplifies the value function of an intertemporal optimization (infinite dimensional) problem associated with a convex transaction-cost function which depends only on the transactions (velocities) of a commodity evolution: it states that the value function is equal to the marginal fonction of a fin…
Optimizes dividend and reinsurance strategies for correlated insurance lines.
problem Stochastic control of optimal reinsurance and dividend policies for multiple insurance lines.
method Maximizes cumulative discounted dividends using a Hamilton-Jacobi-Bellman equation and finite difference method.
result Provides optimal strategies for transferring risk among reinsurers.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
We consider in this paper the optimal dividend problem for an insurance company whose uncontrolled reserve process evolves as a classical Cramér--Lundberg process. The firm has the option of investing part of the surplus in a Black--Scholes financial market. The objective is to find a strategy consisting of both invest…
We consider a singular control problem with regime switching that arises in problems of optimal investment decisions of cash-constrained firms. The value function is proved to be the unique viscosity solution of the associated Hamilton-Jacobi-Bellman equation. Moreover, we give regularity properties of the value functi…
We study an optimal execution problem in a continuous-time market model that considers market impact. We formulate the problem as a stochastic control problem and investigate properties of the corresponding value function. We find that right-continuity at the time origin is associated with the strength of market impact…
Paper approximates free boundary for optimal investment stopping problems.
problem Optimal investment stopping problems with utility maximization.
method Dual control method to derive asymptotic properties and construct a global closed-form approximation.
result Global closed-form approximation of dual free boundary reduces computational cost.
Bayes-CPACE optimally explores continuous BAMDPs.
problem Model uncertainty in continuous state and action spaces.
method Covering state-belief-action space with samples, exploiting Lipschitz continuity.
result Near-optimal value function computed efficiently.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
We consider the classical optimal dividends problem under the Cramér-Lundberg model with exponential claim sizes subject to a constraint on the time of ruin. We introduce the dual problem and show that the complementary slackness conditions are satisfied, thus there is no duality gap. Therefore the optimal value functi…
We consider the problem of learning the optimal action-value function in the discounted-reward Markov decision processes (MDPs). We prove a new PAC bound on the sample-complexity of model-based value iteration algorithm in the presence of the generative model, which indicates that for an MDP with N state-action pairs a…
We characterize value functions in partially observable MDPs as semi-algebraic sets.
problem Understanding feasible value functions in partially observable Markov decision processes.
method Characterization of feasible value functions as semi-algebraic sets defined by polynomial inequalities.
result The feasible set of value functions in POMDPs is a semi-algebraic set, not a polytope as in MDPs.
We give a new proof of the fact that the value function of the finite time horizon American put option for a jump diffusion, when the jumps are from a compound Poisson process, is the classical solution of a free boundary equation. We also show that the value function is C1 across the optimal stopping boundary. Our …
Direct method solves complex option pricing in changing market conditions.
problem Pricing financial options with changing market conditions.
method Reduces complex optimal stopping problems to simpler ones, finding explicit value functions.
result Simpler and more direct solution for option pricing in regime-switching models.
This paper is a continuation of Ishitani and Kato (2015), in which we derived a continuous-time value function corresponding to an optimal execution problem with uncertain market impact as the limit of a discrete-time value function. Here, we investigate some properties of the derived value function. In particular, we …
Paper tackles robust control of SDEs with ambiguity, proving value function existence and applying to investment problems.
problem Robust control of SDEs with ambiguity parameters and non-Lipschitz coefficients.
method Existence and uniqueness of value function established through BSDEs with non-linear growth conditions.
result Existence and uniqueness of value function in proper space, verified through BSDEs.
Estimates personalized treatment response curves using covariates.
problem Flexible estimation of personalized treatment response curves.
method Sieve based nonparametric estimator of smoothed regimen-response curve function.
result Asymptotic linearity and undersmoothing criteria for efficient estimation.
The value function of an optimal stopping problem for jump diffusions is known to be a generalized solution of a variational inequality. Assuming that the diffusion component of the process is nondegenerate and a mild assumption on the singularity of the Lévy measure, this paper shows that the value function of this op…
Kernel smoothing improves LLM reasoning efficiency.
problem Efficiently estimate value functions with limited samples for reinforcement learning.
method Kernelized advantage estimation using classical nonparametric statistics.
result Improved policy optimization with accurate value and gradient estimation.
We study the optimal liquidation problem in a market model where the bid price follows a geometric pure jump process whose local characteristics are driven by an unobservable finite-state Markov chain and by the liquidation rate. This model is consistent with stylized facts of high frequency data such as the discrete n…
Study optimal liquidation strategies with infinite horizon and regime switching.
problem Optimal liquidation with semimartingale strategies in a stochastic environment.
method Characterization of value function and optimal strategy via BSDEs with infinite horizon.
result Existence and uniqueness of optimal control problem solutions.
We consider the problem of portfolio optimization in a simple incomplete market and under a general utility function. By working with the associated Hamilton-Jacobi-Bellman partial differential equation (HJB PDE), we obtain a closed-form formula for a trading strategy which approximates the optimal trading strategy whe…
Optimizing dividend payments for an insurance company with bounded rates.
problem Maximizing expected exponential utility of discounted dividends under bounded dividend rates.
method Suboptimal strategies are evaluated using a new method to estimate the distance to the value function.
result The optimal strategy is of barrier type with a non-linear barrier.
We develop a polynomial method to optimize trading in markets with transaction costs.
problem Optimizing trading strategies in markets with proportional transaction costs.
method Polynomial approximation of the residual value function to determine optimal trading strategies.
result Identify the trade-off between trading frequency and trade sizes for satisfactory agreement with theoretically optimal strategies.