On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,982 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “soft Q-learning”

Pretrain Soft Q-Learning with Imperfect Demonstrationscs.LG

Paper proposes a pretraining method for soft Q-learning from imperfect demonstrations.

problem Challenges in exploiting expert demonstrations while maintaining exploration potentials.

method γ-discounted biased policy evaluation with entropy regularization.

result Our method effectively learns from imperfect demonstrations and outperforms other methods.

reinforcement learning soft Q-learning demonstrations policy evaluation actor-critic

2019-05-09abs ↗pdf ↗

Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamicsstat.ML

This paper analyzes how periodic and soft target updates stabilize linear Q-learning.

problem Theoretical explanation of stabilization mechanisms for linear Q-learning.

method Exact analysis using switched linear system dynamics and the joint spectral radius.

result Periodic and soft target updates can guarantee convergence to the exact projected Q-Bellman solution under specific conditions.

Q-learning linear function approximation stabilization periodic updates soft updates

2026-05-31abs ↗pdf ↗

Composable Deep Reinforcement Learning for Robotic Manipulationcs.LG

Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world ro…

2018-03-19abs ↗pdf ↗

Boosting Soft Q-Learning by Boundingcs.LG

Boosts Q-learning by using value function bounds.

problem Efficiently solving new tasks using past experience.

method Derives double-sided bounds on optimal value function and uses them to update Q-function.

result Boosted training performance through alternative Q-function update method.

Q-learning value function bounds performance boost

2024-06-26abs ↗pdf ↗

On Gaussian approximation for entropy-regularized Q-learning with function approximationstat.ML

The paper analyzes the convergence rates of Q-learning with entropy regularization and linear function approximation.

problem Analyzing the convergence rates of Q-learning with entropy regularization and linear function approximation.

method The paper derives rates of convergence using the high-dimensional central limit theorem, linearization of the soft Bellman recursion, and Gaussian approximation for the leading martingale term.

result The algorithm's last iterate satisfies high-order moment bounds, with a Gaussian approximation bound of order

n^{-1/4}

Q-learning entropy regularization linear function approximation convergence rates Gaussian approximation

2026-05-17abs ↗pdf ↗

Neural Temporal-Difference and Q-Learning Provably Converge to Global Optimacs.LG

Neural TD and Q-learning prove to converge globally to optimal solutions.

problem Nonconvexity and divergence in neural TD due to value function approximation.

method Proving global convergence of neural TD and Q-learning using overparametrization of neural networks.

result Neural TD and Q-learning converge globally to the global optimum of mean-squared projected Bellman error.

reinforcement learning temporal-difference learning neural networks global convergence policy evaluation

2019-05-24abs ↗pdf ↗

Discriminator Soft Actor Critic without Extrinsic Rewardscs.LG

DSAC improves robustness of imitation learning.

problem Learning from few expert data and distribution shift.

method DSAC uses adversarial reward function.

result DSAC performs well on PyBullet environments.

imitation learning reinforcement learning soft actor critic adversarial reward distribution shift

2020-01-19abs ↗pdf ↗

Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learningcs.LG

A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.

problem Stabilizing model-free off-policy deep reinforcement learning with soft divergence.

method Representing past experiences as a QGraph, selecting a subgraph with favorable structure, and using lower bounds for temporal difference learning.

result QG-DDPG method is less prone to soft divergence and more robust to hyperparameters.

deep reinforcement learning model-free off-policy QGraph temporal difference

2020-07-15abs ↗pdf ↗

Potential-Based Advice for Stochastic Policy Learningcs.LG

The paper uses potential functions to help reinforcement learning agents learn optimal policies.

problem Learning optimal stochastic policies in reinforcement learning.

method Augmenting the reward with potential functions and applying these to policy gradient algorithms.

result Potential-based reward shaping preserves optimality of stochastic policies and speeds up learning.

reinforcement learning policy learning potential functions stochastic policies policy gradient

2019-07-20abs ↗pdf ↗

Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecs.LG

New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.

problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.

method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with

L^2

-convergence guarantees.

result PAC guarantees for offline RL under partial coverage and realizability conditions.

offline reinforcement learning minimax optimization soft Q-function partial coverage realizability

2023-02-05abs ↗pdf ↗

Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theorycs.LG

Temporal-difference and Q-learning learn feature representations that converge to optimal ones.

problem Understanding how feature representations evolve in temporal-difference and Q-learning with neural networks.

method Mean-field theory applied to overparameterized two-layer neural networks.

result The feature representation converges to the optimal one, generalizing previous results.

temporal-difference Q-learning feature representation mean-field theory overparameterized neural networks

2020-06-08abs ↗pdf ↗

Causal Deep Reinforcement Learning Using Observational Datacs.LG

Paper proposes methods to use observational data for reinforcement learning, addressing confounding issues.

problem Using observational data for reinforcement learning can lead to misleading outcomes due to unobserved confounders.

method The paper introduces two deconfounding methods in deep reinforcement learning to adjust for confounders.

result The proposed deconfounding methods improve the accuracy of reinforcement learning models using observational data.

reinforcement learning observational data deconfounding causal inference deep Q-learning

2022-11-28abs ↗pdf ↗

Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximationcs.LG

Algorithm learns Nash equilibria in stochastic games using entropy-regularized policies.

problem Learning Nash equilibria in zero-sum stochastic games is computationally expensive.

method Entropy-regularized soft policies for Q-function updates.

result Algorithm converges to Nash equilibrium under certain conditions.

Nash equilibrium zero-sum games policy approximation Q-learning entropy regularization

2020-09-01abs ↗pdf ↗

Adversarial Learning of a Sampler Based on an Unnormalized Distributioncs.LG

We investigate adversarial learning in the case when only an unnormalized form of the density can be accessed, rather than samples. With insights so garnered, adversarial learning is extended to the case for which one has access to an unnormalized form u(x) of the target density function, but no samples. Further, new c…

2019-01-03abs ↗pdf ↗

Bridging the Gap Between Value and Policy Based Reinforcement Learningcs.AI

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy pro…

2017-02-28abs ↗pdf ↗

SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcs.LG

SUNRISE improves off-policy RL algorithms by integrating ensemble methods.

problem Stability and exploration issues in off-policy RL algorithms.

method SUNRISE combines ensemble-based weighted Bellman backups and upper-confidence bounds for efficient exploration.

result SUNRISE improves the performance of off-policy RL algorithms across various domains.

reinforcement learning ensemble methods off-policy deep learning exploration

2020-07-09abs ↗pdf ↗

On the Convergence of Approximate and Regularized Policy Iteration Schemesstat.ML

Paper analyzes convergence of entropy-regularized reinforcement learning methods.

problem Ensuring sub-optimality control in entropy-regularized RL.

method Unified analysis of regularized MPI and VI schemes, providing convergence rates.

result Established sufficient conditions for convergence and provided explicit rates.

reinforcement learning entropy regularization policy iteration value iteration convergence rates

2019-09-20abs ↗pdf ↗

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcs.LG

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

2018-01-04abs ↗pdf ↗

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcs.LG

Enhances SAC for better sample efficiency in continuous-action tasks.

problem Improving sample efficiency in soft actor-critic algorithms.

method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).

result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.

Soft Actor-Critic Reinforcement Learning Sample Efficiency Priority Experience Replay Continuous-Action Tasks

2019-06-10abs ↗pdf ↗

A Theory of Regularized Markov Decision Processescs.LG

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and…

2019-01-31abs ↗pdf ↗

The Mean-Squared Error of Double Q-Learningcs.LG

Double Q-learning has the same mean-squared error as Q-learning under certain conditions.

problem Comparing the mean-squared error of Double Q-learning and Q-learning.

method Theoretical analysis based on Lyapunov equations for both tabular and linear function approximation settings.

result The asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning under specific conditions.

mean-squared error Double Q-learning Q-learning Lyapunov equations asymptotic analysis

2020-07-09abs ↗pdf ↗

Imitating Language via Scalable Inverse Reinforcement Learningcs.LG

New approach uses inverse reinforcement learning to improve language model training.

problem Training large language models using imitation learning methods.

method Developed a new method of inverse reinforcement learning to optimize sequences directly.

result IRL-based fine-tuning leads to better performance and diversity in language generation.

language models reinforcement learning imitation learning inverse Q-learning temporal difference

2024-09-02abs ↗pdf ↗

Learning Hard Alignments with Variational Inferencecs.AI

There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition. Hard attention can offer benefits over soft attention such as decreased computational cost, but training hard attention models can be difficult because of the discrete la…

2017-05-16abs ↗pdf ↗

A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcs.LG

Paper analyzes Q-learning with neural networks, proving a fast convergence rate.

problem Analyzing the convergence rate of neural Q-learning.

method Finite-time analysis of neural Q-learning with a deep ReLU network.

result Neural Q-learning converges to optimal policy with

O(1/\sqrt{T})

rate.

Q-learning neural networks finite-time analysis convergence rate Markov decision process

2019-12-10abs ↗pdf ↗

Q-Learning with Differential Entropy of Q-Tablescs.LG

DE-QT detects optimal Q-learning stopping points.

problem Information loss in Q-learning during prolonged training.

method Introducing DE-QT to detect entropy changes in Q-tables.

result DE-QT identifies the best stopping point for Q-learning.

Q-learning entropy information loss policy search differential entropy

2020-06-26abs ↗pdf ↗

Momentum Q-learning with Finite-Sample Convergence Guaranteecs.LG

This paper analyzes momentum Q-learning with finite-sample guarantees.

problem Improving Q-learning performance with momentum schemes.

method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.

result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.

Q-learning momentum finite-sample analysis function approximation convergence rate

2020-07-30abs ↗pdf ↗

Exploratory Control with Tsallis Entropy for Latent Factor Modelsq-fin.MF

Optimal control in latent factor models uses Tsallis entropy for exploration.

problem Optimal control in models with latent factors.

method Reward exploration with Tsallis entropy and derive

q

-Gaussian distribution over states.

result Optimal policy derived in a model-agnostic setting.

optimal control latent factors Tsallis entropy exploration q-Gaussian

2022-11-14abs ↗pdf ↗

Ensemble Bootstrapping for Q-Learningcs.LG

EBQL reduces bias in Q-learning for improved performance.

problem Over- and under-estimation biases in Q-learning degrade performance.

method Ensemble Bootstrapping to reduce both over- and under-estimation biases.

result EBQL outperforms other Q-learning methods in Atari games.

Q-learning reinforcement learning bias reduction deep Q-learning ATARI games

2021-02-28abs ↗pdf ↗

Periodic Q-Learningcs.LG

PQ-learning improves Q-learning by periodically updating target estimates.

problem Improving sample complexity in Q-learning for finding optimal policies.

method Maintains two Q-value estimates, one online and one target, updated periodically.

result PQ-learning achieves better sample complexity for finding epsilon-optimal policies.

Q-learning target networks sample complexity Markov decision processes

2020-02-23abs ↗pdf ↗

Self-Imitation Learning via Generalized Lower Bound Q-learningcs.LG

New self-imitation learning method improves performance in continuous control tasks.

problem Improving off-policy learning in continuous control tasks.

method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.

result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.

self-imitation learning lower-bound Q-learning continuous control n-step Q-learning performance improvement

2020-06-12abs ↗pdf ↗

Linear

Q

-Learning Does Not Diverge in

L^2

: Convergence Rates to a Bounded Setcs.LG

Linear Q-learning converges to a bounded set without divergence.

problem Proving linear Q-learning does not diverge and converges to a bounded set.

method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.

result First

L^2

convergence rate of linear Q-learning iterates to a bounded set.

Q-learning linear function approximation convergence rate bounded set ε-softmax policy

2025-01-31abs ↗pdf ↗

On the Complexity of Offline Reinforcement Learning with

Q^\star

-Approximation and Partial Coveragecs.LG

Study shows offline RL under $Q^\star$ -approximation and partial coverage is harder than previously thought.

problem Theoretical limits of offline reinforcement learning under

Q^\star

-approximation and partial coverage.

method Introduced a decision-estimation framework to decompose offline RL complexity into decision and value estimation errors.

result Answered the open question by proving sample inefficiency under partial coverage is not guaranteed by

Q^\star

-realizability and Bellman completeness.

offline reinforcement learning partial coverage information theory decision-estimation Bellman completeness

2026-02-12abs ↗pdf ↗

Performing Deep Recurrent Double Q-Learning for Atari Gamescs.LG

Proposes a new Deep Recurrent Double Q-Learning model for Atari games.

problem Improving Deep Reinforcement Learning models for Atari games.

method Integrates Double Q-Learning and Recurrent Networks (LSTM, DRQN) for Atari games.

result Demonstrates improved performance in Atari games compared to existing models.

Deep Reinforcement Learning Atari Games Deep Recurrent Q-Learning Double Q-Learning Recurrent Networks

2019-08-16abs ↗pdf ↗

Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysisstat.ML

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

problem Understanding the sample complexity of Q-learning in synchronous settings.

method Analyzing Q-learning in synchronous MDPs with state and action spaces, proving minimax optimal sample complexity for TD learning and Q-learning under certain conditions.

result Q-learning requires more samples than minimax bounds suggest, revealing strict sub-optimality when action space is more than one.

Q-learning Markov decision process sample complexity minimax bounds TD learning

2021-02-12abs ↗pdf ↗

Concentration bounds for SSP Q-learning for average cost MDPscs.LG

Q-learning for average cost MDPs gets a concentration bound.

problem Finding bounds for Q-learning in average cost MDPs.

method Derives a concentration bound using shortest path problem equivalence.

result Numerical comparison with relative value iteration shows the bound's effectiveness.

Q-learning Markov decision processes concentration bounds shortest path problem

2022-06-07abs ↗pdf ↗

Whittle index based Q-learning for restless bandits with average rewardcs.LG

New algorithm uses Whittle index to improve Q-learning for restless bandits.

problem Optimizing decision-making in multiarmed restless bandits with average reward.

method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.

result Demonstrates significant computational gains and excellent empirical performance.

Whittle index restless bandits Q-learning reinforcement learning average reward

2020-04-29abs ↗pdf ↗

On the Reduction of Variance and Overestimation of Deep Q-Learningcs.LG

New dropout technique reduces variance and overestimation in deep Q-Learning.

problem Reduction of variance and overestimation in deep Q-Learning.

method Using Dropout techniques to reduce variance and overestimation in deep Q-Learning.

result Demonstrated effectiveness in enhancing stability and reducing both variance and overestimation.

Deep Q-Learning Dropout Variance Reduction Overestimation Reinforcement Learning

2019-10-14abs ↗pdf ↗

Lookahead-Bounded Q-Learningcs.LG

LBQL improves Q-learning by using lookahead bounds for better performance.

problem Improving Q-learning in stochastic environments.

method LBQL uses lookahead bounds to construct dual penalties and track upper and lower bounds via stochastic approximation.

result LBQL converges faster and is more robust to hyperparameters than standard Q-learning.

Q-learning lookahead bounds stochastic environments convergence robustness

2020-06-28abs ↗pdf ↗

Provably Efficient Kernelized Q-Learningcs.LG

Kernelized Q-learning achieves good performance with minimal data.

problem Efficient Q-learning in high-dimensional spaces.

method Kernelized Q-learning framework with effective dimensionality.

result Concrete regret bounds for linear and Gaussian RBF kernels.

Q-learning kernel methods regret bounds effective dimensionality control tasks

2022-04-21abs ↗pdf ↗

On the Estimation Bias in Double Q-Learningcs.LG

Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.

problem Underestimation bias in double Q-learning leading to non-optimal fixed points.

method Proposes a simple approach using approximate dynamic programming to bound the target value.

result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.

double Q-learning underestimation bias overestimation bias approximate dynamic programming non-optimal fixed points

2021-09-29abs ↗pdf ↗

Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimizationcs.LG

Improved Q-learning for robot control with faster data-efficiency.

problem Poor data-efficiency and stochasticity in deep Q-learning.

method Composite Q-learning: Decomposes Q-function into short- and long-term predictions.

result Significant learning speedup and improved data-efficiency in robot tasks.

Q-learning robot control temporal-difference data-efficiency stochasticity

2019-09-30abs ↗pdf ↗

The Sample-Communication Complexity Trade-off in Federated Q-Learningcs.LG

New algorithm reduces sample and communication complexities in federated Q-learning.

problem Optimal Q-function learning in federated Q-learning with limited communication.

method Introduced Fed-DVR-Q algorithm for order-optimal sample and communication complexities.

result Complete characterization of sample-communication complexity trade-off.

federated Q-learning sample complexity communication complexity discount factor order optimality

2024-08-30abs ↗pdf ↗

Accelerated Target Updates for Q-learningcs.LG

Accelerates Q-learning by updating targets more efficiently.

problem Improving convergence speed in Q-learning algorithms.

method Proposes an accelerated target update scheme based on momentum methods.

result The proposed algorithms converge faster than vanilla Q-learning.

Q-learning acceleration reinforcement learning convergence

2019-05-07abs ↗pdf ↗

Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decompositionstat.ML

This paper improves Q-learning bounds using reference-advantage decomposition.

problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.

method Develops a novel error decomposition framework to prove gap-dependent regret bounds.

result Establishes logarithmic gap-dependent regret bounds for Q-learning.

Q-learning MDPs gap-dependent bounds reference-advantage decomposition policy switching cost

2024-10-10abs ↗pdf ↗

Whittle Index Learning Algorithms for Restless Bandits with Constant Stepsizescs.LG

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.

method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.

result The algorithms learn the Whittle index effectively.

Whittle index restless bandits Q-learning constant stepsizes DQN

2024-09-06abs ↗pdf ↗

Counterfactual Survival Q-learning via Buckley-James Boosting, with Applications to ACTG 175 and CALGB 8923stat.ML

Proposes a new Q-learning method for survival outcomes in clinical trials.

problem Incomplete follow-up data and nonlinear covariate effects in clinical trials.

method Combines Buckley-James boosting with flexible base learners for estimating optimal treatment regimes.

result Improves treatment decision accuracy and stability in longitudinal clinical trials.

Q-learning survival analysis clinical trials counterfactual Buckley-James

2025-08-14abs ↗pdf ↗

Is Q-Learning Provably Efficient? An Extended Analysiscs.LG

Extended analysis of Q-learning's efficiency, matching optimal regret.

problem Theoretical guarantees of Q-learning's efficiency and optimal regret.

method Survey of related research, detailed proof reasoning.

result Q-learning with UCB exploration achieves sample efficiency matching optimal regret.

Q-learning reinforcement learning UCB exploration theoretical guarantees optimal regret

2020-09-22abs ↗pdf ↗

Q-Learning in enormous action spaces via amortized approximate maximizationcs.LG

AQL uses amortized inference to handle high-dimensional action spaces in Q-learning.

problem Difficulty in maximizing over large action spaces in Q-learning.

method Replace expensive maximization over all actions with a maximization over a small subset sampled from a learned proposal distribution.

result AQL outperforms existing methods on continuous control tasks with up to 21 dimensional actions.

Q-learning amortized inference action spaces continuous control maximization

2020-01-22abs ↗pdf ↗