Proposes a new Deep Recurrent Double Q-Learning model for Atari games.
problem Improving Deep Reinforcement Learning models for Atari games.
method Integrates Double Q-Learning and Recurrent Networks (LSTM, DRQN) for Atari games.
result Demonstrates improved performance in Atari games compared to existing models.
This paper concerns automated vehicles negotiating with other vehicles, typically human driven, in crossings with the goal to find a decision algorithm by learning typical behaviors of other vehicles. The vehicle observes distance and speed of vehicles on the intersecting road and use a policy that adapts its speed alo…
New dropout technique reduces variance and overestimation in deep Q-Learning.
problem Reduction of variance and overestimation in deep Q-Learning.
method Using Dropout techniques to reduce variance and overestimation in deep Q-Learning.
result Demonstrated effectiveness in enhancing stability and reducing both variance and overestimation.
Deep Q-learning generates directed acyclic graphs.
problem Generating DAGs with specified structures.
method Deep reinforcement learning, specifically deep Q-learning.
result Demonstrated capability of generating DAGs in sparse reward environments.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
Theoretical analysis improves understanding of Deep Q-Learning's behavior.
problem Lack of formal guarantees and gaps between theory and practice of Deep Q-Learning.
method Dynamical systems perspective, focusing on realistic assumptions.
result Proves convergence of Deep Q-Learning under specific conditions.
Proposes a deep spectral Q-learning for mobile health data.
problem Personalized treatment assignment for patients with time-varying covariates.
method Integrates PCA with deep Q-learning for mixed frequency data.
result Mean return converges to optimal under estimated optimal policy.
EBQL reduces bias in Q-learning for improved performance.
problem Over- and under-estimation biases in Q-learning degrade performance.
method Ensemble Bootstrapping to reduce both over- and under-estimation biases.
result EBQL outperforms other Q-learning methods in Atari games.
Reverse Experience Replay improves Deep Q-learning for sparse rewards.
problem Sparse rewards and reward-maximizing tasks in Deep Q-learning.
method Sampling transitions in reverse order for training.
result Significantly increased performance in tasks with limited experience and memory capacity.
Paper analyzes Q-learning with neural networks, proving a fast convergence rate.
problem Analyzing the convergence rate of neural Q-learning.
method Finite-time analysis of neural Q-learning with a deep ReLU network.
result Neural Q-learning converges to optimal policy with O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) rate. PQ-learning improves Q-learning by periodically updating target estimates.
problem Improving sample complexity in Q-learning for finding optimal policies.
method Maintains two Q-value estimates, one online and one target, updated periodically.
result PQ-learning achieves better sample complexity for finding epsilon-optimal policies.
Deep Q-learning analyzes EEG for drowsiness during driving tests.
problem Estimating drowsiness during driving to improve safety.
method Adapting Q-learning to EEG data for drowsiness estimation.
result Trained model accurately tracks mind state variations in EEG data.
A new algorithm SRG-DQN reduces variance in deep Q-learning.
problem Inaccurate estimation of anchor points in SVRG for deep Q-learning.
method Introduces recursive gradient variance reduction for stochastic gradient updates.
result Demonstrates improved efficiency and effectiveness of SRG-DQN on reinforcement learning tasks.
ConQUR tackles delusional bias in deep Q-learning, improving performance in Atari games.
problem Delusional bias in deep Q-learning.
method Efficient methods to mitigate delusional bias by training Q-approximators with consistent labels and a search framework.
result Improves performance in Atari games, sometimes dramatically.
Low-complexity spiking networks learn complex tasks with minimal trainable parameters.
problem Training complex reinforcement learning tasks with minimal resources.
method Reinforcement learning on simple networks of spiking neurons with random connections.
result Small random spiking networks achieve learning efficiency similar to humans on complex tasks.
Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitiv…
A new algorithm for deep Q-learning with robustness to state transition uncertainty.
problem Model uncertainty in state transitions for non-tabular, continuous state spaces.
method Distributionally robust approach using worst-case transition ball and dualized Bellman operator with Sinkhorn distance.
result Optimal policy found through solving non-linear Bellman equation with neural network parameterization.
Deep Q-learning optimizes same-day delivery with vehicles and drones.
problem Optimizing same-day delivery with limited vehicle and drone capacities.
method Deep Q-learning approach to assign packages to vehicles or drones.
result Deep Q-learning policy outperforms benchmark policies and maintains effectiveness with changing fleet sizes.
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
problem Overestimation and high variance in Q-Learning cause learning algorithms to diverge in complex environments.
method Deep Weighted Q-Learning (Deep WQL) uses Dropout and Monte Carlo sampling to approximate WQL's weights and reduce bias.
result Deep WQL reduces bias and improves performance on benchmarks compared to existing methods.
Improved Q-learning for robot control with faster data-efficiency.
problem Poor data-efficiency and stochasticity in deep Q-learning.
method Composite Q-learning: Decomposes Q-function into short- and long-term predictions.
result Significant learning speedup and improved data-efficiency in robot tasks.
Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood,…
Paper presents a new method for Bayesian deep learning that scales to Atari games.
problem Training neural networks on complex environments like Atari games is challenging.
method Adapted temporal difference Q-learning to work with Bayesian inference.
result TAGI allows for analytical inference of neural network parameters, achieving performance comparable to gradient-based methods.
Recent advances in deep reinforcement learning have achieved human-level performance on a variety of real-world applications. However, the current algorithms still suffer from poor gradient estimation with excessive variance, resulting in unstable training and poor sample efficiency. In our paper, we proposed an innova…
A controller learns to control a nonlinear plant with unknown model and partial observation using continuous deep Q-learning.
problem Designing a controller for a nonlinear plant with unknown model and partial sensor observation under network delays.
method Continuous deep Q-learning applied to an extended state including past control inputs and outputs.
result The controller can learn a robust control policy to network delays with partial sensor observation.
In this article, we sketch an algorithm that extends the Q-learning algorithms to the continuous action space domain. Our method is based on the discretization of the action space. Despite the commonly used discretization methods, our method does not increase the discretized problem dimensionality exponentially. We wil…
Kernelized Q-learning achieves good performance with minimal data.
problem Efficient Q-learning in high-dimensional spaces.
method Kernelized Q-learning framework with effective dimensionality.
result Concrete regret bounds for linear and Gaussian RBF kernels.
Online random forests improve Q-learning performance in specific gym environments.
problem Improving Q-learning performance in reinforcement learning tasks.
method Proposed online random forests as Q-function approximators and growing them as learning progresses.
result Improved performance over state-of-the-art Deep Q-Networks in specific gym environments.
Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world ro…
New algorithms speed up inverse reinforcement learning by solving MDPs once.
problem Slow convergence in Maximum Entropy Inverse Reinforcement Learning.
method Deep Inverse Q-learning with constraints exploiting Q-learning.
result Up to several orders of magnitude speedup compared to existing methods.
Deep Q-Learning optimizes market making by balancing price risk and spread profits.
problem Optimizing liquidity provision in financial markets.
method Reinforcement Learning applied to a market making problem with a reward function.
result Deep Q-Learning algorithms can recover the optimal market making strategy.
Tabular Q-learning outperforms advanced RL methods in monetary policy.
problem Dynamic setting of short-term interest rates to stabilize inflation and unemployment under uncertain macroeconomic conditions.
method Discrete-action Markov Decision Process with tabular Q-learning, SARSA, Actor-Critic, Deep Q-Networks, Bayesian Q-learning, POMDP formulations.
result Standard tabular Q-learning achieved the best performance (-615.13 +- 309.58 mean return) compared to advanced RL methods and traditional policy rules.
Deep Q-learning is investigated as an end-to-end solution to estimate the optimal strategies for acting on time series input. Experiments are conducted on two idealized trading games. 1) Univariate: the only input is a wave-like price time series, and 2) Bivariate: the input includes a random stepwise price time series…
Stochastic Q-learning tackles large action spaces with reduced computation.
problem Effective decision-making in complex environments with large discrete action spaces.
method Stochastic value-based RL approaches that consider a sublinear number of actions in each iteration.
result Stochastic Q-learning achieves near-optimal returns with significantly reduced computation time.
A Deep Q-Learning framework tackles market-making by incorporating closing auctions.
problem Managing end-of-day risk in market-making models.
method Developed a Deep Q-Learning framework that anticipates closing auctions and continuously refines projected clearing prices.
result The Deep Q-Learning framework outperforms classical market-making models in simulations and real data.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
The paper investigates the effectiveness of reusing experience in Deep Q-Learning for FPS environments.
problem The high number of interactions required for reinforcement learning limits its practicality.
method The authors test the effectiveness of applying learning update steps multiple times per environmental step in the VizDoom environment.
result Updating learning steps less frequently than every 4th environmental step does not improve performance and can degrade performance.
A new approach to hedging using contextual bandit models outperforms traditional methods.
problem Effective replication of financial contracts in incomplete markets with low transaction costs.
method Viewing hedging as a contextual k k k -armed bandit problem, using reinforcement learning. result The contextual bandit model provides more accurate and sample-efficient hedging than Q Q Q -learning. We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outp…
A deep Q-learning strategy optimizes portfolio trading efficiency.
problem Optimizing dynamic portfolio allocation schemes.
method Formulated a Markov decision process model with deep Q-learning for discrete combinatorial actions.
result Outperforms benchmark strategies in real-world trading simulations.
A new Q-learning variant reduces underestimation bias in deep reinforcement learning.
problem Underestimation bias in deep reinforcement learning policies.
method Introducing a novel, parameter-free Deep Q-learning variant.
result Significantly outperforms existing approaches and improves state-of-the-art performance.
We introduce a novel Deep Reinforcement Learning (DRL) algorithm called Deep Quality-Value (DQV) Learning. DQV uses temporal-difference learning to train a Value neural network and uses this network for training a second Quality-value network that learns to estimate state-action values. We first test DQV's update rules…
The paper formalizes and analyzes multi-agent Q-learning with value factorization.
problem Understanding and improving the convergence of multi-agent Q-learning with value factorization.
method Formalized a multi-agent fitted Q-iteration framework for analyzing factorized multi-agent Q-learning.
result Multi-agent Q-learning with linear value factorization can converge under certain conditions.
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as …
Q( Δ Δ Δ )-Learning improves Q-Learning by separating action-value functions into different time scales.
problem Q-Learning struggles with bias-variance trade-off, especially in long-term rewards.
method Introduces Q( Δ Δ Δ )-Learning, extending TD( Δ Δ Δ ) to decompose Q( Δ Δ Δ )-function into distinct discount factors. result Q( Δ Δ Δ )-Learning achieves better stability and scalability, especially for long-term tasks. CAQL tackles continuous action maximization in Q-learning.
problem Maximizing continuous actions in Q-learning.
method Developed CAQL algorithms using plug-and-play optimizers and MIP for optimal max-Q.
result CAQL outperforms policy-based methods in heavily constrained environments.
Deep reinforcement learning finds optimal learning policies for adaptive systems.
problem Finding individualized learning plans for learners with unknown latent traits.
method Formulated as a Markov decision process, applied deep Q-learning with a transition model estimator.
result The algorithm efficiently discovers optimal learning policies with small data sets.
Deep Q-Learning models optimal exercise strategies for option-type products.
problem Modeling optimal exercise strategies for option-type products.
method Reinforcement learning approach using deep neural networks to approximate the Q-function.
result Pricing the contract at inception and deriving bounds on the option price.