A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
We approximate derivatives of functions on manifolds by embedding them and applying vector-valued operators.
Paper develops efficient RL algorithm for general value function approximation.
Develops hierarchical reinforcement learning value function approximators.
In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low d…
A core operation in reinforcement learning (RL) is finding an action that is optimal with respect to a learned value function. This operation is often challenging when the learned value function takes continuous actions as input. We introduce deep radial-basis value functions (RBVFs): value functions learned using a de…
Recently, Petrik et al. demonstrated that L1Regularized Approximate Linear Programming (RALP) could produce value functions and policies which compared favorably to established linear value function approximation techniques like LSPI. RALP's success primarily stems from the ability to solve the feature selection and va…
We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local traj…
A large proportion of market making models derive from the seminal model of Avellaneda and Stoikov. The numerical approximation of the value function and the optimal quotes in these models remains a challenge when the number of assets is large. In this article, we propose closed-form approximations for the value functi…
We consider the problem of portfolio optimization in a simple incomplete market and under a general utility function. By working with the associated Hamilton-Jacobi-Bellman partial differential equation (HJB PDE), we obtain a closed-form formula for a trading strategy which approximates the optimal trading strategy whe…
In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…
Efficiently plans large MDPs with weak function approximations.
We consider a singular control problem with regime switching that arises in problems of optimal investment decisions of cash-constrained firms. The value function is proved to be the unique viscosity solution of the associated Hamilton-Jacobi-Bellman equation. Moreover, we give regularity properties of the value functi…
In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable function…
New algorithms accelerate value function convergence in MDPs.
Offline RL struggles with sample efficiency due to fundamental barriers.
In this paper we study a utility maximization problem with both optimal control and optimal stopping in a finite time horizon. The value function can be characterized by a variational equation that involves a free boundary problem of a fully nonlinear partial differential equation. Using the dual control method, we der…
Randomized value functions offer a promising approach towards the challenge of efficient exploration in complex environments with high dimensional state and action spaces. Unlike traditional point estimate methods, randomized value functions maintain a posterior distribution over action-space values. This prevents the …
Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function . Two novel algorithms are proposed to approximate the true value…
A novel approach uses an ensemble of Gaussian processes for robust and adaptive reinforcement learning.
We study the finite horizon Merton portfolio optimization problem in a general local-stochastic volatility setting. Using model coefficient expansion techniques, we derive approximations for the both the value function and the optimal investment strategy. We also analyze the `implied Sharpe ratio' and derive a series a…
Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.
Tackling large approximate dynamic programming or reinforcement learning problems requires methods that can exploit regularities, or intrinsic structure, of the problem in hand. Most current methods are geared towards exploiting the regularities of either the value function or the policy. We introduce a general classif…
Boosts Q-learning by using value function bounds.
Extends Tanimoto kernel to real-valued functions.
We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation t…
Bayesian reinforcement learning (BRL) offers a decision-theoretic solution for reinforcement learning. While "model-based" BRL algorithms have focused either on maintaining a posterior distribution on models or value functions and combining this with approximate dynamic programming or tree search, previous Bayesian "mo…
This paper studies optimal approximation factors in misspecified off-policy RL, identifying key factors under various settings.
We develop a polynomial method to optimize trading in markets with transaction costs.
We present a fully nonparametric method to estimate the value function, via simulation, in the context of expected infinite-horizon discounted rewards for Markov chains. Estimating such value functions plays an important role in approximate dynamic programming and applied probability in general. We incorporate "soft in…
Deep Galerkin Method estimates value function for mean-field control problem.
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
Value-function approximation methods that operate in batch mode have foundational importance to reinforcement learning (RL). Finite sample guarantees for these methods often crucially rely on two types of assumptions: (1) mild distribution shift, and (2) representation conditions that are stronger than realizability. H…
Framework optimizes battery storage for markets by separating long-term degradation from short-term market dynamics.
A new RL paradigm reduces state-action-value function approximation inefficiency.
Bayesian approach improves Shapley value estimation efficiency.
This paper develops numerical methods for finding optimal dividend pay-out and reinsurance policies. A generalized singular control formulation of surplus and discounted payoff function are introduced, where the surplus is modeled by a regime-switching process subject to both regular and singular controls. To approxima…
New method for off-policy evaluation in POMDPs using future-dependent value functions.
New algorithm explores reinforcement learning with noisy data.
In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving ve…
Kernel smoothing improves LLM reasoning efficiency.
Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as ne…
Model-free reinforcement learning algorithms combined with value function approximation have recently achieved impressive performance in a variety of application domains. However, the theoretical understanding of such algorithms is limited, and existing results are largely focused on episodic or discounted Markov decis…
Kernel-based function approximation improves reinforcement learning performance.
In this paper, we provide two new stable online algorithms for the problem of prediction in reinforcement learning, \emph{i.e.}, estimating the value function of a model-free Markov reward process using the linear function approximation architecture and with memory and computation costs scaling quadratically in the siz…
We assume a continuous-time price impact model similar to Almgren-Chriss but with the added assumption that the price impact parameters are stochastic processes modeled as correlated scalar Markov diffusions. In this setting, we develop trading strategies for a trader who desires to liquidate his inventory but faces pr…
In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…
New algorithm FLUTE achieves uniform-PAC convergence in RL with linear approx.