This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
Unified framework for CO problems using RL, providing optimal solutions and convergence guarantees.
problem Combinatorial optimization problems
method Unified framework of Markov decision processes (MDPs) and value-based reinforcement learning (RL) techniques
result RL techniques converge to approximate solutions with a guarantee on optimality gap
Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some glo…
New RL algorithms adapt to time limits, improving task performance.
problem Fixed RL behaviors cannot adapt to different time restrictions.
method Introduced two algorithms for time adaptive RL: Independent Gamma-Ensemble and n-Step Ensemble.
result Zero-shot adaptation between different time restrictions.
Stochastic Q-learning tackles large action spaces with reduced computation.
problem Effective decision-making in complex environments with large discrete action spaces.
method Stochastic value-based RL approaches that consider a sublinear number of actions in each iteration.
result Stochastic Q-learning achieves near-optimal returns with significantly reduced computation time.
New RL algorithm handles delayed feedback with posterior sampling.
problem Challenges of delayed feedback in reinforcement learning with linear function approximation.
method Posterior sampling with delayed feedback for value-based RL.
result Achieves optimal regret guarantee with improved computational efficiency.
Reincarnating RL reuses prior work to accelerate RL progress.
problem Efficiency and accessibility in reinforcement learning for large-scale applications.
method Transfer of learned policies between RL agents or design iterations, focusing on value-based RL.
result Demonstrated gains in performance over tabula rasa RL on various tasks.
This paper shows using classification instead of regression improves deep RL scalability.
problem Challenges in training value functions for large networks in deep RL.
method Used categorical cross-entropy loss instead of mean squared error regression.
result Significant improvements in performance and scalability across various domains.
RL teaches agents to make decisions through trial and error.
problem Sequential decision-making problems in dynamic environments.
method Value-based and policy search approaches, including actor-critic methods.
result RL can learn good policies from numeric reward feedback.
The paper explores how different loss functions impact reinforcement learning algorithms.
problem Improving reinforcement learning algorithms by optimizing loss functions.
method Comprehensive survey on loss functions in reinforcement learning, proving the benefits of specific loss functions.
result Binary cross-entropy loss leads to first-order bounds and is more efficient than squared loss.
Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that…
We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper…
RL tackles decision making in unknown environments, focusing on efficiency and efficacy.
problem Efficiency and efficacy in RL algorithms for sample-starved situations.
method Markov Decision Processes, model-based and value-based approaches, policy optimization.
result Enhanced understanding and improvements in sample and computational efficacies of RL algorithms.
Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied only with respect to (worst-case) approximation error, in the more classical approximate dynamic pro…
Privileged Information Dropout improves RL performance without distillation.
problem Improving sample efficiency and performance in reinforcement learning.
method Introducing Privileged Information Dropout to directly incorporate privileged information into RL agent inputs.
result Privileged Information Dropout outperforms distillation and auxiliary tasks in a partially-observed environment.
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal poli…
RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
Under-parameterization hinders deep RL's efficiency.
problem Implicit under-parameterization impairs data-efficiency in deep RL.
method Characterized and mitigated the rank collapse of value network features.
result Controlling rank collapse improves deep RL performance.
SUNRISE improves off-policy RL algorithms by integrating ensemble methods.
problem Stability and exploration issues in off-policy RL algorithms.
method SUNRISE combines ensemble-based weighted Bellman backups and upper-confidence bounds for efficient exploration.
result SUNRISE improves the performance of off-policy RL algorithms across various domains.
New GFlowNet training framework using policy gradients for combinatorial object generation.
problem Training efficiency and robustness in GFlowNet models.
method Policy-dependent rewards and coupled training strategy for forward and backward policies.
result Advanced RL perspectives for robust gradient estimation improve GFlowNet performance.
New RL method learns from state transitions without actions.
problem Offline RL with missing action labels.
method State policy discretisation and decQN algorithm.
result Improves convergence speed and performance in online RL.
In this work, we consider the problem of model selection for deep reinforcement learning (RL) in real-world environments. Typically, the performance of deep RL algorithms is evaluated via on-policy interactions with the target environment. However, comparing models in a real-world environment for the purposes of early …
Survey of RL in finance, tackling complex decision-making.
problem Complex financial decision-making problems with limited model assumptions.
method Value and policy-based RL algorithms, neural networks, deep RL.
result Improved financial decision-making with less model assumptions.
New method improves policies by combining Markov and non-Markov strategies.
problem Improving policies in reinforcement learning.
method Geometric Policy Composition (GPI) using a geometric horizon model (GHM).
result GPI can improve non-Markov policies by composing GHMs of base Markov policies.
New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.
problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.
method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with L2-convergence guarantees. result PAC guarantees for offline RL under partial coverage and realizability conditions.
Deep RL controls anesthesia more accurately than traditional methods.
problem Controlling the level of unconsciousness during anesthesia.
method Deep Reinforcement Learning (DRL) to map patient state to propofol dosage.
result Deep RL model outperformed traditional controllers (1.7% vs 3.4% median absolute performance error).
Most practical recommender systems focus on estimating immediate user engagement without considering the long-term effects of recommendations on user behavior. Reinforcement learning (RL) methods offer the potential to optimize recommendations for long-term user engagement. However, since users are often presented with…
While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL. In particular, it remains elusive how to design a provably efficient policy optimization algorithm that incorporates exploration. To bridge su…
Paper introduces efficient online sub-sampling for RL with function approximation, reducing policy updates.
problem Efficiently managing computation complexity in RL with general function approximation.
method Online sub-sampling framework that measures information gain and guides exploration.
result Policy updates reduced to ∝polylog(K) times for near-optimal regret. Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algor…
New RL algorithm tackles non-stationary environments with flexible policy updates.
problem Non-stationary reinforcement learning with time-varying rewards and transition probabilities.
method Model-free policy-based algorithm NS-NAC with restart-based exploration and dynamic learning rates.
result Dynamic regret of ildeO(∣S∣1/2∣A∣1/2ΔT1/6T5/6) for both algorithms. Financial portfolio management is one of the problems that are most frequently encountered in the investment industry. Nevertheless, it is not widely recognized that both Kelly Criterion and Risk Parity collapse into Mean Variance under some conditions, which implies that a universal solution to the portfolio optimizat…
BCPO optimizes offline RL policies by converting uncertainty into conservative bounds.
problem Offline RL's fragility under distribution shifts and model errors.
method Bayesian approach with credible lower bounds and KL regularization.
result BCPO yields an uncertainty-calibrated policy that avoids exploiting model errors.
We study reinforcement learning (RL) in high dimensional episodic Markov decision processes (MDP). We consider value-based RL when the optimal Q-value is a linear function of d-dimensional state-action feature representation. For instance, in deep-Q networks (DQN), the Q-value is a linear function of the feature repres…
DRL agents perform poorly at high decision frequencies, but a new algorithm improves performance.
problem DRL agents struggle at high decision frequencies, leading to poor performance.
method Proved that DRL agents' action-conditioned return distributions collapse to their policy's return distribution as decision frequency increases. Defined superiority as a probabilistic generalization of advantage for high-frequency value-based RL.
result Proper modeling of superiority distribution improves performance of controllers at high decision frequencies.
DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.
problem Policy optimization in RL under distribution shift and adversarial dynamics.
method DR-RPO algorithm incorporating reference-policy regularization and upper confidence bonus for exploration.
result DR-RPO achieves sublinear regret and polynomial suboptimality bounds in robust RL.
This paper proposes a new approach to RL by focusing on the value-improvement path.
problem Value prediction problems in RL are sequence-dependent and require holistic approach.
method Characterize and approximate the value-improvement path holistically.
result A representation that spans the value-improvement path provides accurate value approximations for future policy improvements.
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributiona…
Paper tackles sample-efficient RL for linearly realizable MDPs with limited revisiting.
problem Sample-efficient reinforcement learning for linearly realizable MDPs with limited revisiting.
method Develops a new sampling protocol that allows for backtracking and revisiting states in a controlled manner.
result Achieves polynomial sample complexity scaling with feature dimension, horizon, and inverse sub-optimality gap.
Mixreg improves RL generalization by mixing diverse training environments.
problem Overfitting in RL agents trained in limited environments.
method Trains on a mixture of diverse observations and imposes linearity constraints.
result Mixreg outperforms baselines on unseen testing environments.
This work explores efficient reinforcement learning with density features in low-rank MDPs.
problem Efficient reinforcement learning with density features in low-rank MDPs.
method Proposes algorithms for off-policy estimation and online construction of exploratory data distributions.
result Demonstrates sample-efficient learning with density features in low-rank MDPs, overcoming technical challenges.
This paper proposes a new optimization objective for value-based deep reinforcement learning. We extend conventional Deep Q-Networks (DQNs) by adding a model-learning component yielding a transcoder network. The prediction errors for the model are included in the basic DQN loss as additional regularizers. This augmente…
Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple similar tas…
EarnHFT tackles HFT challenges with hierarchical RL, significantly outperforming existing methods.
problem Challenges in applying RL to HFT due to long trajectories and market volatility.
method Three-stage hierarchical RL framework: Q-teacher, diverse RL agents, and minute-level router.
result Significantly outperforms 6 state-of-the-art baselines in profitability.
New RL method improves financial index tracking accuracy.
problem Optimizing financial index tracking with dynamic market information.
method Discrete-time dynamic model, Banach fixed point iteration, deep reinforcement learning.
result Proposed RL method outperforms benchmarks in tracking accuracy.
New issue found in value-based reinforcement learning for stochastic environments.
problem Value-based reinforcement learning struggles with stochastic state transitions.
method Demonstrated using a multiobjective Markov Decision Process (MOMDP).
result Approaches may converge to Pareto-dominated solutions instead of optimal ones.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.