On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Q-function regularization”

q-Learning in Continuous Timecs.LG

Continuous-time Q-learning theory developed for reinforcement learning.

problem Continuous-time reinforcement learning challenges.

method Entropy-regularized, exploratory diffusion process formulation; first-order approximation of Q-function; martingale conditions.

result Developed a q-learning theory independent of time discretization.

reinforcement learning continuous time Q-learning entropy regularization martingales

2022-07-02abs ↗pdf ↗

Continuous-time q-learning for mean-field control problemscs.LG

This paper develops q-learning methods for mean-field control problems.

problem Continuous-time mean-field control problems with interaction between agents.

method Introduces two q-functions and devises model-free learning algorithms.

result Developed algorithms can learn optimal value functions and q-functions.

q-learning mean-field control reinforcement learning continuous time martingale condition

2023-06-28abs ↗pdf ↗

Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecs.LG

New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.

problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.

method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with

L^2

-convergence guarantees.

result PAC guarantees for offline RL under partial coverage and realizability conditions.

offline reinforcement learning minimax optimization soft Q-function partial coverage realizability

2023-02-05abs ↗pdf ↗

Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximationcs.LG

Algorithm learns Nash equilibria in stochastic games using entropy-regularized policies.

problem Learning Nash equilibria in zero-sum stochastic games is computationally expensive.

method Entropy-regularized soft policies for Q-function updates.

result Algorithm converges to Nash equilibrium under certain conditions.

Nash equilibrium zero-sum games policy approximation Q-learning entropy regularization

2020-09-01abs ↗pdf ↗

Manifold Regularization for Kernelized LSTDcs.LG

Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL). It is a necessary component of policy iteration and can be used for variance reduction in policy gradient methods. Therefore its quality has a significant impact on most RL algorithms. Motivated by manifol…

2017-10-15abs ↗pdf ↗

Logistic Q-Learningcs.LG

Proposes a new reinforcement learning algorithm using Q-function.

problem Optimal control in Markov Decision Processes (MDPs).

method Regularized linear-programming formulation, Q-function, saddle-point optimization.

result Demonstrates effectiveness on various benchmark problems.

reinforcement learning optimal control MDPs Q-function saddle-point optimization

2020-10-21abs ↗pdf ↗

GRAC: Self-Guided and Self-Regularized Actor-Criticcs.LG

GRAC improves reinforcement learning by self-guiding and self-regularizing.

problem Learning divergence and slow updates in reinforcement learning algorithms.

method Self-regularized TD-learning and self-guided policy improvement.

result Achieved or outperformed state-of-the-art results on OpenAI gym tasks.

reinforcement learning actor-critic self-regularization self-guiding deep learning

2020-09-18abs ↗pdf ↗

Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimationcs.LG

A new algorithm reduces sample complexity for learning Q-functions in reinforcement learning.

problem Efficiently learning Q-functions in reinforcement learning with continuous state and action spaces.

method Developed a simple, iterative learning algorithm that estimates low-rank Q-functions.

result Achieved exponential improvement in sample complexity for low-rank Q-functions.

reinforcement learning Q-function sample efficiency low-rank matrix estimation kernel methods

2020-06-11abs ↗pdf ↗

Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functionscs.LG

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

problem Gradient ascent in complex Q-functions leads to suboptimal solutions.

method SAVO actor generates multiple action proposals and truncates poor local optima.

result SAVO actor finds optimal actions more frequently and outperforms other architectures.

reinforcement learning actor-critic DDPG TD3 local optima

2024-10-15abs ↗pdf ↗

Minimum information divergence of Q-functions for dynamic treatment resumesstat.ME

The paper introduces a new method for estimating optimal policies in dynamic treatment regimes using information geometry.

problem Estimating optimal policies in dynamic treatment regimes.

method Minimum information divergence method based on

γ

-power divergence.

result The

γ

-power divergence method effectively seeks the optimal policy by vanishing the divergence between policy-equivalent Q-functions.

reinforcement learning dynamic treatment regimes information geometry Q-function policy equivalence

2022-11-16abs ↗pdf ↗

On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationmath.ST

Study on $Q$ -function estimation for continuous state-action MDPs, deriving rates and conditions.

problem Estimating

Q

-function in off-policy evaluation for continuous state-action Markov decision processes.

method Reformulated as nonparametric instrumental variables (NPIV) problem, derived minimax lower bounds, proposed sieve two-stage least squares estimator.

result First minimax lower bounds for

Q

-function and its derivatives in sup-norm and

L^2

-norm, same as classical nonparametric regression.

Q-function estimation off-policy evaluation nonparametric instrumental variables Markov decision processes minimax lower bounds

2022-01-17abs ↗pdf ↗

SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcs.LG

SPQR improves Q-ensemble diversity in reinforcement learning.

problem Overestimation bias in Q-learning for complex tasks.

method Introduces SPQR for Q-ensemble independence regularization.

result SPQR outperforms baseline algorithms in online and offline RL benchmarks.

reinforcement learning Q-learning ensemble methods overestimation bias random matrix theory

2024-01-06abs ↗pdf ↗

Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learningcs.LG

Entropy regularization improves policy optimization in reinforcement learning.

problem Improving policy optimization in reinforcement learning.

method Entropy regularization is introduced to soften the greedy policy towards a more diverse softmax policy, leading to a continuously parameterized algorithm that interpolates between policy gradient and Q-learning.

result An intermediate algorithm can improve performance in reinforcement learning.

reinforcement learning policy optimization entropy regularization policy gradient Q-learning

2020-05-18abs ↗pdf ↗

Conservative Q-Learning for Offline Reinforcement Learningcs.LG

CQL learns conservative Q-functions to improve offline RL performance.

problem Leveraging large, static datasets in reinforcement learning without further interaction.

method Conservative Q-learning (CQL) which learns a conservative Q-function to lower-bound policy values.

result CQL substantially outperforms existing offline RL methods, often achieving 2-5 times higher final returns.

reinforcement learning offline RL conservative Q-learning policy learning value lower-bound

2020-06-08abs ↗pdf ↗

Target Transfer Q-Learning and Its Convergence Analysiscs.LG

Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…

2018-09-21abs ↗pdf ↗

CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learningcs.LG

CCLF improves sample efficiency in RL by prioritizing informative samples.

problem Learning from high-dimensional observations is challenging and unstable.

method Contrastive-Curiosity-Driven Learning Framework (CCLF) that prioritizes informative samples.

result CCLF improves sample efficiency and learning performance in RL.

reinforcement learning sample efficiency curiosity-driven contrastive learning Q-function regularization

2022-05-02abs ↗pdf ↗

Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inferencestat.ML

Improves DRL for long-term causal inference with semiparametric methods.

problem Efficient inference for policy values in nonparametric MDPs with stringent conditions.

method Semiparametric Double Reinforcement Learning (DRL) with superefficient nonparametric estimators.

result Relaxes overlap conditions and reduces high-dimensional density-ratio estimation.

Double Reinforcement Learning Long-term Causal Inference Semiparametric Methods Q-function Occupancy Density Ratios

2025-01-12abs ↗pdf ↗

Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescs.LG

Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the…

2018-07-16abs ↗pdf ↗

Preventing Value Function Collapse in Ensemble {Q}-Learning by Maximizing Representation Diversitycs.LG

New technique prevents Q-learning collapse by maximizing diversity among ensembles.

problem Value function collapse in ensemble Q-learning.

method Maximizing representation diversity through regularization.

result Regularized approach significantly outperforms existing methods.

Q-learning ensemble representation diversity regularization maximization

2020-06-24abs ↗pdf ↗

Harnessing Structures for Value-Based Planning and Reinforcement Learningcs.LG

Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some glo…

2019-09-26abs ↗pdf ↗

On Instrumental Variable Regression for Deep Offline Policy Evaluationcs.LG

This paper explores how IV methods can improve Q-function estimates in offline policy evaluation.

problem Confounding in estimating Q-function using reinforcement learning.

method Integrates IV techniques into offline policy evaluation (OPE) to improve Q-function estimates.

result State-of-the-art OPE methods are closely matched in performance by some IV methods.

Instrumental Variables Offline Policy Evaluation Q-function Estimation Reinforcement Learning Confounding

2021-05-21abs ↗pdf ↗

Hidden Markov Model Estimation-Based Q-learning for Partially Observable Markov Decision Processcs.LG

The objective is to study an on-line Hidden Markov model (HMM) estimation-based Q-learning algorithm for partially observable Markov decision process (POMDP) on finite state and action sets. When the full state observation is available, Q-learning finds the optimal action-value function given the current action (Q func…

2018-09-17abs ↗pdf ↗

Robust Temporal Difference Learning for Critical Domainscs.LG

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the $κ$ -operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We i…

2019-01-23abs ↗pdf ↗

Policy Optimization Achieves Data-Dependent Regret Bounds in MDPs with Unknown Transitionscs.LG

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

problem Achieving best-of-both-worlds guarantees with data-dependent regret bounds in MDPs with unknown transitions.

method Optimistic follow-the-regularized-leader algorithm with new optimistic Q-function estimators and transition bonus.

result First-order, second-order, and path-length bounds with polylog(T) regret in the stochastic regime.

policy optimization Markov decision processes unknown transitions regret bounds optimistic algorithms

2026-06-30abs ↗pdf ↗

Provable Offline Reinforcement Learning for Structured Cyclic MDPsstat.ML

CycleFQI tackles offline reinforcement learning for cyclic MDPs, mitigating state distribution mismatch.

problem Offline reinforcement learning for cyclic MDPs with heterogeneous dynamics and discount factors.

method CycleFQI decomposes the cyclic process into stage-wise sub-problems, using vector of stage-specific Q-functions.

result CycleFQI mitigates the curse of dimensionality and provides finite-sample suboptimality error bounds.

reinforcement learning cyclic MDPs offline learning Q-functions state distribution mismatch

2026-02-12abs ↗pdf ↗

Convergence of Q-value in case of Gaussian rewardsmath.OC

The paper proves the convergence of Q-value for Gaussian rewards.

problem Existing proofs cannot guarantee convergence of the Q-function for Gaussian rewards.

method Using the central limit theorem and relaxing the condition to

E[r(s,a)^2]<\infty

result Proves the convergence of the Q-function under the condition of

E[r(s,a)^2]<\infty

reinforcement learning Q-value Gaussian rewards central limit theorem policy gradient theorem

2020-03-07abs ↗pdf ↗

Fast Non-Episodic Finite-Horizon RL with K-Step Lookahead Thresholdingcs.LG

New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.

problem Challenges in online reinforcement learning for non-episodic, finite-horizon MDPs.

method Introduces a K-step lookahead Q-function with a time-varying threshold for selecting actions.

result Achieves minimax optimal constant regret for K=1 and

\mathcal{O}(\max((K-1),C_{K-1})\sqrt{SAT\log(T)})

regret for K ≥ 2.

reinforcement learning finite horizon lookahead Q-function thresholding tabular learning

2026-01-31abs ↗pdf ↗

On the Expressivity of Neural Networks for Deep Reinforcement Learningcs.LG

We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$ -functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$ -func…

2019-10-14abs ↗pdf ↗

Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systemscs.LG

Proposes a RL method using simulators for stabilizing uncertain systems.

problem Limited experiences and potential dangerous actions during RL learning of real systems.

method Two-stage approach: virtual systems for Q-function learning, real system interactions for final policy.

result Proposed method improves RL performance in uncertain discrete-time systems.

RL Stabilization Simulators Uncertain Systems Deep Q-Learning

2021-01-13abs ↗pdf ↗

Structured Difference-of-Q via Orthogonal Learningstat.ML

New method estimates and optimizes policy differences using orthogonal learning.

problem Offline reinforcement learning with safety concerns and cost limitations.

method Dynamic R-learner for estimating and optimizing

Q^π(s,1)-Q^π(s,0)

, leveraging orthogonal estimation.

result Consistent policy optimization with improved convergence rates.

reinforcement learning offline learning policy optimization orthogonal learning causal inference

2024-06-12abs ↗pdf ↗

Q-learning for Optimal Control of Continuous-time Systemseess.SY

In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based…

2014-10-11abs ↗pdf ↗

Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approachcs.LG

New approach transfers rewards learned in one environment to reinforcement learning in a new environment.

problem Transfer of rewards learned using inverse reinforcement learning from one environment to a new, different environment.

method Formulate the problem as a joint system of Bellman equations, develop minimax estimators for the target soft-

q

-function, solve the source and target system of equations jointly.

result The coupled approach removes the first-order influence of source Bellman residual error compared to the sequential approach.

reinforcement learning inverse reinforcement learning reward transfer minimax approach Bellman equations

2026-05-27abs ↗pdf ↗

Universal Approximation Theorem of Deep Q-Networkscs.LG

DQNs can approximate optimal Q-functions with high accuracy on compact sets.

problem Approximating optimal Q-functions in continuous-time Markov Decision Processes.

method Stochastic control, FBSDEs, residual network approximation theorems, large deviation bounds, viscosity solutions.

result DQNs can approximate optimal Q-functions on compact sets with arbitrary accuracy and high probability.

Deep Q-Networks Approximation Theorem Continuous-Time Markov Decision Process Forward-Backward Stochastic Differential Equations Residual Network

2025-05-04abs ↗pdf ↗

Revisiting the Softmax Bellman Operator: New Benefits and New Perspectivecs.LG

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

2018-12-02abs ↗pdf ↗

Learning to Communicate Using Counterfactual Reasoningcs.LG

MACC learns communication protocols by adapting counterfactual reasoning.

problem Credit assignment and non-stationarity in communication environments.

method Adapts counterfactual reasoning to overcome credit assignment and uses action policy and Q-function of other agents to handle non-stationarity.

result MACC outperforms state-of-the-art baselines in four scenarios.

multi-agent reinforcement learning counterfactual reasoning communication protocols credit assignment non-stationarity

2020-06-12abs ↗pdf ↗

Solving optimal stopping problems with Deep Q-Learningq-fin.PR

Deep Q-Learning models optimal exercise strategies for option-type products.

problem Modeling optimal exercise strategies for option-type products.

method Reinforcement learning approach using deep neural networks to approximate the Q-function.

result Pricing the contract at inception and deriving bounds on the option price.

Deep Q-Learning Optimal stopping Option pricing Reinforcement learning Neural networks

2021-01-24abs ↗pdf ↗

Zeroth-order Deterministic Policy Gradientcs.LG

ZDPG learns model-free policies without critics, improving on PG.

problem Model-free policy learning in complex dynamic problems.

method Approximates policy-reward gradients via two-point stochastic evaluations of the Q-function.

result Restores true model-free policy learning without critics, with improved stability and efficiency.

Zeroth-order Deterministic Policy Gradient Model-free Policy learning Stochastic evaluations

2020-06-12abs ↗pdf ↗

Learning Nearly Decomposable Value Functions Via Communication Minimizationcs.LG

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning full…

2019-10-11abs ↗pdf ↗

Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimizationcs.LG

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…

2019-09-30abs ↗pdf ↗

SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learningcs.LG

SF-DQN improves RL transfer by learning successor features.

problem Transfer RL with shared dynamics but different reward functions.

method Decomposes Q-function into SF and reward mapping; uses GPI for policy improvement.

result SF-DQN with GPI converges faster and generalizes better than traditional RL methods.

Reinforcement Learning Transfer Learning Successor Features Policy Improvement Deep Neural Networks

2024-05-24abs ↗pdf ↗

A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Modelscs.LG

A new method reduces complexity in estimating dynamic choice models.

problem Estimating structural parameters in dynamic discrete choice models using behavioral data.

method Two-stage approach: inverse reinforcement learning for Q-function estimation, state selection via clustering, and maximum likelihood estimation with nested fixed-point algorithm.

result The method mitigates the curse of dimensionality and provides finite-sample bounds on estimation error.

dynamic discrete choice state aggregation inverse reinforcement learning maximum likelihood estimation curse of dimensionality

2023-04-11abs ↗pdf ↗

Quadratic Q-network for Learning Continuous Control for Autonomous Vehiclescs.LG

A hybrid model combines Q-learning and PID controller for continuous vehicle control.

problem Learning unsatisfactory results with discrete action space in autonomous driving.

method Combining Q-learning and PID controller, using Quadratic Q-function approximation and action network.

result Autonomous vehicle successfully learns smooth and efficient driving behavior.

Reinforcement Learning Autonomous Driving Continuous Control PID Controller Quadratic Q-function

2019-11-29abs ↗pdf ↗

Actor-Free Continuous Control via Structurally Maximizable Q-Functionscs.LG

Proposes a value-based method for continuous control without an actor.

problem Computational infeasibility of evaluating Q-values in continuous action spaces.

method Structurally maximizable Q-functions, actor-free approach.

result Performance and sample efficiency comparable to actor-critic methods.

reinforcement learning continuous control value-based actor-free Q-function

2025-10-21abs ↗pdf ↗

What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescs.AI

Explains agent behavior through intended outcomes in reinforcement learning.

problem Proving impossibility of general post-hoc explanations in reinforcement learning.

method Derives local explanations based on intention for Q-function approximations, proving consistency with learned Q-values.

result Demonstrates the necessity of collecting information during training for accurate explanations.

Reinforcement Learning Intended Outcomes Q-function Explanations Consistency

2020-11-10abs ↗pdf ↗

Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluationcs.LG

A variety of machine learning models have been proposed to assess the performance of players in professional sports. However, they have only a limited ability to model how player performance depends on the game context. This paper proposes a new approach to capturing game context: we apply Deep Reinforcement Learning (…

2018-05-26abs ↗pdf ↗

di-Langlands correspondence and extended observableshep-th

New quantum integrals discovered for a spin chain model.

problem Exploring quantum integrals for a spin chain model.

method Using surface defects and observables in 4D

{\mathcal{N}}=2

super-QCD.

result First construction of quantum integrals and their joint eigenvectors.

Yangian spin chain surface defects quantum integrals Q-operators

2024-02-21abs ↗pdf ↗

Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweepingcs.LG

New algorithm learns efficiently in multi-agent settings.

problem Efficient learning in multi-agent Markov decision processes.

method Cooperative Prioritized Sweeping: model-based reinforcement learning with sample efficiency.

result Outperforms state-of-the-art on SysAdmin and randomized environments.

multi-agent reinforcement learning Cooperative Prioritized Sweeping sample efficiency Markov decision processes temporal difference learning

2020-01-15abs ↗pdf ↗

Revisiting Bellman Errors for Offline Model Selectioncs.LG

New Bellman error estimator improves offline model selection performance.

problem Selecting the best policy from logged data using mean squared Bellman error.

method Developed a more accurate estimator of MSBE and analyzed conditions for successful OMS.

result New estimator achieves impressive offline model selection performance on diverse tasks.

Offline RL Model Selection Bellman Error Q-function Atari Games

2023-01-31abs ↗pdf ↗