We present the use of the fitted Q iteration in algorithmic trading. We show that the fitted Q iteration helps alleviate the dimension problem that the basic Q-learning algorithm faces in application to trading. Furthermore, we introduce a procedure including model fitting and data simulation to enrich training data as…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Fitted Q-Iteration”
We propose a method for efficient training of Q-functions for continuous-state Markov Decision Processes (MDPs) such that the traces of the resulting policies satisfy a given Linear Temporal Logic (LTL) property. LTL, a modal logic, can express a wide range of time-dependent logical properties (including "safety") that…
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
The QLBS model is a discrete-time option hedging and pricing model that is based on Dynamic Programming (DP) and Reinforcement Learning (RL). It combines the famous Q-Learning method for RL with the Black-Scholes (-Merton) model's idea of reducing the problem of option pricing and hedging to the problem of optimal reba…
Study shows high-dimensional sparse RL hardness and Lasso Q-iteration's nearly dimension-free regret.
problem Hardness of online sparse reinforcement learning in high-dimensional MDPs.
method Lower bound construction and Lasso fitted Q-iteration analysis.
result Lasso Q-iteration achieves nearly dimension-free regret of with oracle access to a good exploratory policy.
New method for robust policy evaluation in offline reinforcement learning with sequentially exogenous unobserved confounders.
problem Offline reinforcement learning in domains with unobserved confounders.
method Orthogonalized robust fitted-Q-iteration with closed-form solutions and bias-correction.
result Effective in simulations and real-world data, improving robustness and computational ease.
This work characterizes conditions for offline policy evaluation in reinforcement learning.
problem Understanding when classical methods succeed in offline policy evaluation for linear function approximation.
method Control-theoretic and linear-algebraic conditions for classical methods (FQI and LSTD).
result A precise hierarchy of regimes under which these estimators succeed, and a complete picture of their behavior.
New RL algorithm explains why deep learning works in stochastic environments.
problem Why deep RL algorithms perform well in practice despite using random exploration.
method Introducing SQIRL, an iterative RL algorithm that separates exploration and learning.
result Effective horizon explains why deep RL works in stochastic environments.
This work proposes robust reinforcement learning methods using both offline and online data.
problem Designing robust policies against parameter uncertainties in high-dimensional systems.
method Proposes RPQ for model-free learning with historical data and HyTQ for hybrid learning with both historical and online data.
result Unified analysis and theoretical guarantees for robust optimal policies in high-dimensional systems.
Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcs.LG
Efficient offline reinforcement learning with neural networks using differentiable function approximation.
problem Statistical efficiency of offline reinforcement learning with function approximators.
method Pessimistic fitted Q-learning (PFQL) and differentiable function approximation.
result Provably efficient offline reinforcement learning with differentiable function approximation.
When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely de…
Two new algorithms improve Q* approximation in batch RL with linear error propagation.
problem Improving Q* approximation in batch reinforcement learning.
method Two novel algorithms that estimate Bellman error directly, without quadratic dependence.
result Linear-in-horizon error propagation for batch RL algorithms.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
A new approach to reinforcement learning improves policy performance by adjusting control frequency.
problem Improving reinforcement learning performance by optimizing control frequency.
method Introducing action persistence and a novel algorithm, PFQI, to learn optimal value function at a given persistence.
result PFQI effectively learns optimal value function with action persistence, improving reinforcement learning performance.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
New algorithm improves knowledge transfer in dynamic decision-making.
problem Utilizing data from existing ventures to improve decision-making in new ventures.
method Proposes Transferred Fitted -Iteration algorithm for estimating optimal action-state function .
result Significantly improved final learning error of function.
The paper formalizes and analyzes multi-agent Q-learning with value factorization.
problem Understanding and improving the convergence of multi-agent Q-learning with value factorization.
method Formalized a multi-agent fitted Q-iteration framework for analyzing factorized multi-agent Q-learning.
result Multi-agent Q-learning with linear value factorization can converge under certain conditions.
Proposes RFQI for robust RL using offline data.
problem Learning robust policies in the presence of model uncertainty.
method RFQI algorithm using offline data to learn optimal robust policy.
result RFQI learns near-optimal robust policy under standard assumptions.
CycleFQI tackles offline reinforcement learning for cyclic MDPs, mitigating state distribution mismatch.
problem Offline reinforcement learning for cyclic MDPs with heterogeneous dynamics and discount factors.
method CycleFQI decomposes the cyclic process into stage-wise sub-problems, using vector of stage-specific Q-functions.
result CycleFQI mitigates the curse of dimensionality and provides finite-sample suboptimality error bounds.
This paper optimizes off-policy evaluation in reinforcement learning with function approximation.
problem Estimating cumulative value of a new policy from logged data generated by an unknown policy.
method Regression-based fitted Q iteration method, equivalent to estimating conditional mean embedding of transition operator.
result The method is minimax-optimal, with nearly minimal estimation error.
Improves DRL for long-term causal inference with semiparametric methods.
problem Efficient inference for policy values in nonparametric MDPs with stringent conditions.
method Semiparametric Double Reinforcement Learning (DRL) with superefficient nonparametric estimators.
result Relaxes overlap conditions and reduces high-dimensional density-ratio estimation.
Predicting the outcomes of integrating Unmanned Aerial Systems (UAS) into the National Aerospace (NAS) is a complex problem which is required to be addressed by simulation studies before allowing the routine access of UAS into the NAS. This thesis focuses on providing 2D and 3D simulation frameworks using a game theore…
Paper analyzes faster convergence rates for reinforcement learning from offline data.
problem Analyzing faster convergence rates for reinforcement learning from offline data.
method Fine analysis of reinforcement learning from offline data, providing fast rates for regret convergence.
result The paper provides fast rates for the regret convergence, showing that the level of exponentiation depends on the noise in the decision-making problem.
Understanding Value Decomposition Algorithms in Deep Cooperative Multi-Agent Reinforcement Learningcs.LG
The paper explores when and why value decomposition algorithms work in cooperative multi-agent reinforcement learning.
problem The applicability and convergence properties of value decomposition algorithms in cooperative multi-agent reinforcement learning are unclear.
method The paper introduces decomposable games and proves that applying the multi-agent fitted Q-Iteration algorithm leads to an optimal Q-function in these games.
result The paper offers theoretical insights into when and why value decomposition algorithms converge in cooperative multi-agent reinforcement learning.
Objective: Anemia is a frequent comorbidity in hemodialysis patients that can be successfully treated by administering erythropoiesis-stimulating agents (ESAs). ESAs dosing is currently based on clinical protocols that often do not account for the high inter- and intra-individual variability in the patient's response. …
A framework uses deep reinforcement learning to optimize energy storage in intraday markets.
problem Optimizing energy storage in intraday markets for renewable energy integration.
method Markov Decision Process, asynchronous distributed fitted Q iteration algorithm, artificial trajectories.
result The agent converges to a policy that achieves higher total revenues than the benchmark strategy.
Initial DR studies mainly adopt model predictive control and thus require accurate models of the control problem (e.g., a customer behavior model), which are to a large extent uncertain for the EV scenario. Hence, model-free approaches, especially based on reinforcement learning (RL) are an attractive alternative. In t…
New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
Motivated by the study of -learning algorithms in reinforcement learning, we study a class of stochastic approximation procedures based on operators that satisfy monotonicity and quasi-contractivity conditions with respect to an underlying cone. We prove a general sandwich relation on the iterate error at each time,…
The coefficient of determination, known as , is commonly used as a goodness-of-fit criterion for fitting linear models. is somewhat controversial when fitting nonlinear models, although it may be generalised on a case-by-case basis to deal with specific models such as the logistic model. Assume we are fittin…
The paper studies efficient Hessian fitting methods for stochastic optimization.
problem Efficient Hessian fitting for stochastic optimization.
method Preconditioned Stochastic Gradient Descent (PSGD) method and Lie groups.
result Hessian fitting problem is strongly convex in certain Lie groups.
The Bezier simplex fitting is a novel data modeling technique which exploits geometric structures of data to approximate the Pareto front of multi-objective optimization problems. There are two fitting methods based on different sampling strategies. The inductive skeleton fitting employs a stratified subsampling from e…
We propose a data aggregation-based algorithm with monotonic convergence to a global optimum for a generalized version of the L1-norm error fitting model with an assumption of the fitting function. The proposed algorithm generalizes the recent algorithm in the literature, aggregate and iterative disaggregate (AID), whi…
CTEF fits ellipsoids to noisy data in any dimension.
problem Fitting ellipsoids to noisy data in arbitrary dimensions.
method Uses the Cayley transform to fit ellipsoids.
result CTEF outperforms other methods, especially when data are not uniformly distributed.
Missing values, irregularly collected samples, and multi-resolution signals commonly occur in multivariate time series data, making predictive tasks difficult. These challenges are especially prevalent in the healthcare domain, where patients' vital signs and electronic records are collected at different frequencies an…
Recognising perfect fitsmath.GT
Algorithm decides if pseudo-Anosov flows have perfect fits.
problem Determining if pseudo-Anosov flows have specific asymptotic properties.
method Algorithm based on box decompositions and universal cover analysis.
result Algorithmic decision on pseudo-Anosov flows' perfect fit status.
We study a resource utilization scenario characterized by intrinsic fitness. To describe the growth and organization of different cities, we consider a model for resource utilization where many restaurants compete, as in a game, to attract customers using an iterative learning process. Results for the case of restauran…
Smoothed fitness landscape improves protein optimization.
problem Infeasibility of combinatorially large protein sequence space.
method Formulate protein fitness as a graph signal, smooth using Tikunov regularization, and optimize with Gibbs sampling.
result 2.5 fold fitness improvement over training set.
JAXFit speeds up curve fitting on GPUs.
problem Nonlinear least squares curve fitting problems.
method Trust region method on GPU with automatic differentiation.
result Significantly faster than CPU and other GPU libraries.
Kernel Multigrid accelerates Back-fitting for additive Gaussian Processes.
problem Slow convergence of Back-fitting in training additive Gaussian Processes.
method Kernel Packets (KP) and Sparse Gaussian Process Regression (GPR) to enhance Back-fitting.
result Kernel Multigrid reduces the required iterations to .
The accumulation of individual fitness or wealth is modelled as a population game in which pairs of individuals are recurrently and randomly matched to play a game over a resource. In addition, all individuals have random access to a constant background resource, and their fitness or wealth depreciates over time. For b…
A framework for eliciting utility functions from investor preferences.
problem Hard elicitation of specific utility functions in portfolio selection.
method Preference-fitting method using probability-wealth pairs and PHARA approximation.
result Fitted utility function converges to the optimal one as more data is used.
Closed form formulas for swaption prices in HJM model are derived. These formulas are used for nonparametric fit of deterministic forward volatility. It is demonstrated that this formula and non-parametric fit works very well and can be used to identify arbitrage opportunities
This paper illustrates a procedure for fitting financial data with -stable distributions. After using all the available methods to evaluate the distribution parameters, one can qualitatively select the best estimate and run some goodness-of-fit tests on this estimate, in order to quantitatively assess its quality. I…
Perfectly Fitting CDO Prices Across Tranches: A Theoretical Framework with Efficient Algorithmsq-fin.RM
Paper proposes a perfect-fit model for CDO tranches.
problem Achieving a perfect fit to market prices across all CDO tranches.
method Introduces compatibility levels, derives conditions, constructs copula models.
result Demonstrates efficient verification and construction of perfect-fit models.
Study shows how varying levels of supervision and orthonormality constraints affect generalization errors in subspace fitting.
problem Effects of varying levels of supervision and orthonormality constraints on generalization errors in subspace fitting.
method Flexible family of problems connecting unsupervised and supervised subspace fitting tasks, explored over a supervision-orthonormality plane.
result Generalization errors of subspace fitting problems follow double descent trends as they become more supervised and less orthonormally constrained.
Study presents MMC model for better fitting multiple choice data.
problem Improving accuracy of latent trait estimates in IRT models.
method Fit autoencoders to MMC model, demonstrating better fit than nominal response model.
result MMC model outperforms traditional IRT models in fit.