A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New RL approach infers optimal policies via variational inference.
The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only …
Off-policy reinforcement learning with eligibility traces is challenging because of the discrepancy between target policy and behavior policy. One common approach is to measure the difference between two policies in a probabilistic way, such as importance sampling and tree-backup. However, existing off-policy learning …
A new algorithm reduces training time for distributed machine learning by dynamically assigning backup workers.
Paper proposes a second-order method for faster SVI convergence.
Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algor…
Multi-step methods such as Retrace() and -step -learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures and their evaluations rarely include enough samples to draw statistically significant conclusions about thei…
MB-DQN uses different backup lengths for improved reinforcement learning.
SUNRISE improves off-policy RL algorithms by integrating ensemble methods.
The project intends to model the stability of power system with a deep learning algorithm to the problem, aiming to delay the removal of the fault. The so-called "fail-delay cut-off" refers to the occurrence of N-1 backup protection action on the backbone network of the system, resulting in longer time for the removal …
We provide a dynamic programming principle for stochastic optimal control problems with expectation constraints. A weak formulation, using test functions and a probabilistic relaxation of the constraint, avoids restrictions related to a measurable selection but still implies the Hamilton-Jacobi-Bellman equation in the …
FOCAL tackles offline meta-reinforcement learning with efficient task inference and behavior regularization.
Modeling supply chain disruptions from climate hazards with adaptive firms.
Sequential decision making in the presence of uncertainty and stochastic dynamics gives rise to distributions over state/action trajectories in reinforcement learning (RL) and optimal control problems. This observation has led to a variety of connections between RL and inference in probabilistic graphical models (PGMs)…
The paper proves well-posedness of nonlocal PDEs related to stochastic control problems.
Optimal control theory connects diffusion models to generative modeling.
The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
Reinforcement learning has attracted great attention recently, especially policy gradient algorithms, which have been demonstrated on challenging decision making and control tasks. In this paper, we propose an active multi-step TD algorithm with adaptive stepsizes to learn actor and critic. Specifically, our model cons…
New method stabilizes FQE by reweighting Bellman targets.
A new method calibrates value predictions in offline RL to improve reliability.
This article studies a portfolio optimization problem, where the market consisting of several stocks is modeled by a multi-dimensional jump-diffusion process with age-dependent semi-Markov modulated coefficients. We study risk sensitive portfolio optimization on the finite time horizon. We study the problem by using a …
MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a…
New Bellman error estimator improves offline model selection performance.
The paper explores solutions to the distributional Bellman equation in reinforcement learning.
We obtain the classical Hanner inequalities by the Bellman function method. These inequalities give sharp estimates for the moduli of convexity of Lebesgue spaces. Easy ideas from differential geometry help us to find the Bellman function using neither "magic guesses" nor calculations.
Optimal portfolio tracking with dynamic capital injection into a ratcheting benchmark.
Richard Bellman's Principle of Optimality, formulated in 1957, is the heart of dynamic programming, the mathematical discipline which studies the optimal solution of multi-period decision problems. In this paper, we look at the main trading principles of Jesse Livermore, the legendary stock operator whose method was pu…
Agent learns to navigate uncertain 3D maps using a hybrid planner.
Polynomial-time RL algorithm for constant actions under linear Bellman completeness.
The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…
In this paper we consider long-run risk sensitive average cost impulse control applied to a continuous-time Feller-Markov process. Using the probabilistic approach, we show how to get a solution to a suitable continuous-time Bellman equation and link it with the impulse control problem. The optimal strategy for the und…
We study utility maximization for power utility random fields with and without intermediate consumption in a general semimartingale model with closed portfolio constraints. We show that any optimal strategy leads to a solution of the corresponding Bellman equation. The optimal strategies are described pointwise in term…
BCRL learns a Bellman complete representation for offline RL policy evaluation.
One-step Bellman alignment improves online RL by reducing task mismatch.
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
We propose a new objective function for finite-horizon episodic Markov decision processes that better captures Bellman's principle of optimality, and provide an expression for the gradient of the objective.
Paper introduces dynamic strategies for multi-period investment models.
Paper analyzes convergence of Sinkhorn algorithm for discrete probability measures on torus.
In the landscape of TD algorithms, the Q(, ) algorithm is an algorithm with the ability to perform a multistep backup in an online manner while also successfully unifying the concepts of sampling with using the expectation across all actions for a state. indicates the extent to which sampling is use…
Decouples critic chunk length from policy to improve policy reactivity and performance.
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a s…
Model quantifies uncertainty's impact on European option prices.
This paper studies systematic exploration for reinforcement learning with rich observations and function approximation. We introduce a new model called contextual decision processes, that unifies and generalizes most prior settings. Our first contribution is a complexity measure, the Bellman rank, that we show enables …
Study solves optimal portfolio selection using HJB equation.
Deep Bellman Hedging uses reinforcement learning to optimize financial portfolio hedging.