HyperAgent improves RL exploration in large-scale problems.
problem Efficient exploration in large-scale reinforcement learning problems.
method Hypermodel framework for incremental posterior approximation without conjugacy.
result HyperAgent achieves logarithmic per-step computational complexity and sublinear regret.
TabNAS improves neural architecture search for tabular datasets by rejecting suboptimal architectures.
problem Finding optimal neural architectures for tabular datasets with resource constraints.
method Develops a reinforcement learning controller motivated by rejection sampling to handle resource constraints.
result TabNAS finds better models that obey resource constraints compared to previous methods.
Tabular Q-learning outperforms advanced RL methods in monetary policy.
problem Dynamic setting of short-term interest rates to stabilize inflation and unemployment under uncertain macroeconomic conditions.
method Discrete-action Markov Decision Process with tabular Q-learning, SARSA, Actor-Critic, Deep Q-Networks, Bayesian Q-learning, POMDP formulations.
result Standard tabular Q-learning achieved the best performance (-615.13 +- 309.58 mean return) compared to advanced RL methods and traditional policy rules.
DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
Paper proposes methods to handle missing data in online RL, improving efficiency and uncertainty capture.
problem Missing data in online RL poses challenges due to the need to impute and act at each time step.
method Proposes fully online imputation ensembles and multiple imputation pathways to balance uncertainty and efficiency.
result Preliminary evidence suggests multiple imputation pathways can be a useful framework for simple and efficient online missing data RL methods.
EQO uses a simple bonus term for efficient exploration in tabular RL.
problem Achieving minimax optimal reinforcement learning with practical efficiency.
method Exploration via Quasi-Optimism, employing a bonus proportional to state-action visit count.
result Achieves the sharpest known regret bound for tabular RL.
Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the re…
The smallest eigenvectors of the graph Laplacian are well-known to provide a succinct representation of the geometry of a weighted graph. In reinforcement learning (RL), where the weighted graph may be interpreted as the state transition process induced by a behavior policy acting on the environment, approximating the …
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
New algorithm reduces RL policy optimization gap.
problem Insufficient theoretical understanding of policy optimization methods.
method Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT)
result Achieves nearly minimax optimal policy-based algorithm for tabular RL.
Reduces RL to online learning, improving RL algorithms.
problem Designing efficient reinforcement learning algorithms.
method Reduction from RL to online learning, focusing on regret minimization and function approximation.
result Demonstrates a new RL algorithm with provable performance guarantees.
MADE improves exploration in RL by maximizing deviation from explored regions.
problem Efficient exploration in high-dimensional RL tasks with sparse rewards.
method Proposes a new exploration approach via maximizing the deviation of the occupancy of the next policy from explored regions, adding it as an adaptive regularizer to the RL objective.
result Significantly improves sample efficiency in navigation and locomotion tasks.
This work bridges offline RL and DRL to address distributional shift.
problem Distributional shift in offline RL due to difference in state-action visitation distributions.
method Proposes offline RL algorithms using DRL framework, characterizes sample complexity under single policy concentrability.
result Demonstrates superior performance of proposed algorithms through simulations.
Paper presents a reduction-based framework for conservative bandits and RL with improved lower and upper bounds.
problem Conservative bandits and reinforcement learning problems.
method Reduction technique to calculate necessary and sufficient budget from baseline policy.
result Improved lower and upper bounds for various conservative settings.
New framework uses unsupervised learning for efficient exploration in RL.
problem Efficient exploration in reinforcement learning with rich observations.
method Combines unsupervised learning and no-regret RL algorithms.
result Proves sample complexity for finding near-optimal policies.
Safe exploration in RF-RL doesn't increase sample complexity.
problem Achieving optimal policies with safety constraints in reward-free RL.
method Proposed SWEET framework for tabular and low-rank MDP settings, leveraging truncated value functions.
result Sample complexities match or outperform constraint-free counterparts, proving safety constraints have little impact.
New RL algorithms learn Gittins indices for unknown Markovian states.
problem Learning Gittins indices for unknown Markovian state transitions.
method Tabular (QGI) and Deep RL (DGN) algorithms based on retirement formulation.
result Lower run time, less storage space, better convergence to Gittins index.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
New RL theory reduces sample complexity for mixing MDPs.
problem Optimal sample complexity for reinforcement learning in mixing MDPs.
method Regeneration-type ideas to analyze mixing times.
result Optimal sample complexity depends on mixing time, not just discount factor.
New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.
problem Challenges in online reinforcement learning for non-episodic, finite-horizon MDPs.
method Introduces a K-step lookahead Q-function with a time-varying threshold for selecting actions.
result Achieves minimax optimal constant regret for K=1 and O ( max ( ( K − 1 ) , C K − 1 ) S A T log ( T ) ) \mathcal{O}(\max((K-1),C_{K-1})\sqrt{SAT\log(T)}) O ( max (( K − 1 ) , C K − 1 ) S A T log ( T ) ) regret for K ≥ 2. New RL algorithm ensures stable, replicable policies.
problem Stability and replicability issues in RL algorithms.
method Introduced weak and strong forms of list replicability, developed a novel planning strategy, and tested state reachability.
result Proved efficient tabular RL algorithm with polynomial list complexity.
A new RL framework allows removing user data without affecting performance.
problem Efficiently removing user data from a reinforcement learning model without affecting performance.
method Formulated a ρ ρ ρ -TV-stable RL algorithm for tabular MDPs that supports exact unlearning. result Achieved a nearly minimax optimal regret bound of Ω ( H S A T + S A H / ρ ) Ω(H\sqrt{\!SAT}\! +\! {SAH}/ρ) Ω ( H S A T + S A H / ρ ) for ρ ρ ρ -TV-stable RL algorithms. All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993). In prior work this was mostly explored in small-state RL problems that allowed tabular representati…
Paper proposes a hybrid RL algorithm that combines offline and online data without needing reward info.
problem How to efficiently use online data to improve RL policies using only offline data.
method A three-stage hybrid RL algorithm that uses reward-agnostic exploration and model-based offline RL.
result The hybrid RL algorithm outperforms both pure offline and pure online RL in sample complexity.
PriMORL trains private RL policies on offline data.
problem Private reinforcement learning on offline data.
method PriMORL learns DP models of the environment and optimizes a policy on the penalized private model.
result PriMORL enables training of private RL agents on complex tasks.
Paper shows RLHF can be solved similarly to standard RL.
problem Difficulty of RLHF compared to standard RL.
method Reduction to reward-based RL techniques.
result RLHF can be solved using existing algorithms for reward-based RL.
Distributional reinforcement learning (distributional RL) has seen empirical success in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However, there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we p…
Optimistic RL algorithms are simplified for deep RL with competitive performance.
problem Achieving accurate optimism in model-based RL for large-scale problems.
method Interpreting scalable optimistic model-based algorithms as solving a tractable noise augmented MDP.
result Competitive regret bound of i l d e O ( ∣ S ∣ H ∣ A ∣ T ) ilde{\mathcal{O}}( |\mathcal{S}|H\sqrt{|\mathcal{A}| T } ) i l d e O ( ∣ S ∣ H ∣ A ∣ T ) for Gaussian noise augmentation. Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
New algorithm for RL using mean embeddings of return distributions.
problem Improving reinforcement learning algorithms for dynamic programming.
method Mean embeddings of return distributions, novel algorithms for RL.
result Asymptotic convergence and improved performance in deep RL.
Offline RL algorithms protect privacy while learning from sensitive data.
problem Learning sensitive data in financial, legal, and healthcare applications.
method Differential privacy guarantees for offline RL algorithms.
result Provable prevention of privacy risks with strong learning bounds.
Self-consistent models improve reinforcement learning by aligning predictions with future values.
problem Improving reinforcement learning by aligning model predictions with future values.
method Proposes multiple self-consistency updates to encourage a learned model and value function to be consistent with each other.
result Self-consistency helps both policy evaluation and control in both tabular and function approximation settings.
Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it ta…
Consider mutli-goal tasks that involve static environments and dynamic goals. Examples of such tasks, such as goal-directed navigation and pick-and-place in robotics, abound. Two types of Reinforcement Learning (RL) algorithms are used for such tasks: model-free or model-based. Each of these approaches has limitations.…
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular …
New offline RL algorithm with optimal sample complexity using LP and error bounds.
problem Finding optimal policies from offline data with limited coverage and function approximation.
method Developed a new LP reformulation with error bounds and constraints for offline RL.
result Achieved optimal O ( 1 / n ) O(1/\sqrt{n}) O ( 1/ n ) sample complexity under various assumptions. New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
New algorithm reduces offline RL data requirements significantly.
problem Optimizing policies using only historical data in reinforcement learning.
method Off-Policy Double Variance Reduction (OPDVR) algorithm.
result OPDVR achieves optimal sample complexity with O ( H 2 / d m ε 2 ) O(H^2/d_mε^2) O ( H 2 / d m ε 2 ) episodes. Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the rew…
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
New BE dimension measure reveals rich RL problems with sample-efficient algorithms.
problem Finding sample-efficient algorithms for complex RL problems.
method Introducing Bellman Eluder (BE) dimension and designing GOLF and OLIVE algorithms.
result GOLF and OLIVE algorithms learn near-optimal policies for low BE dimension problems with polynomial samples.
Quantum RL algorithm achieves logarithmic regret for exploration.
problem Designing efficient quantum RL algorithms for exploration.
method UCRL-style quantum algorithm with lazy updating and quantum estimation.
result Proves O ( p o l y ( S , A , H , log T ) ) \mathcal{O}(\mathrm{poly}(S, A, H, \log T)) O ( poly ( S , A , H , log T )) worst-case regret. Deep RL controls anesthesia more accurately than traditional methods.
problem Controlling the level of unconsciousness during anesthesia.
method Deep Reinforcement Learning (DRL) to map patient state to propofol dosage.
result Deep RL model outperformed traditional controllers (1.7% vs 3.4% median absolute performance error).
Improved RL algorithm for robustness against parameter mismatches.
problem Learning robust control policies against parameter mismatches between training and testing environments.
method Formulated as DR-RL problem, proposed RPVL algorithm for tabular episodic learning with four divergences.
result Achieved i l d e O ( ∣ S ∣ ∣ A ∣ H 5 ) ilde{\mathcal{O}}(|\mathcal{S}||\mathcal{A}| H^{5}) i l d e O ( ∣ S ∣∣ A ∣ H 5 ) sample complexity uniformly better than existing results. New algorithm finds Nash equilibrium in multi-agent games.
problem Extending RL theory to multi-agent settings with large state spaces.
method Proposes an algorithm using an exploiter to find Nash equilibrium policies.
result Proves finding Nash equilibrium is possible with polynomial samples.
New RL algorithm maximizes CVaR in low-rank MDPs with provable efficiency.
problem Maximizing CVaR in large state spaces with function approximation.
method Upper Confidence Bound (UCB) bonus-driven algorithm for low-rank MDPs.
result Achieves sample complexity of O(H^7 A^2 d^4 / τ^2 ε^2) for ε-optimal CVaR.
New algorithm for reward-free RL with linear function approximation, reducing sample complexity.
problem Efficiently learning optimal policies without prior reward information in complex environments.
method Developed an algorithm for reward-free RL in linear Markov decision processes, proving sample complexity bounds.
result Polynomial sample complexity in feature dimension and planning horizon, independent of states and actions.