New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a i l d e O ( d 3 / 4 H 2 K 3 / 4 ) ilde{\mathcal{O}}(d^{3/4}H^2K^{3/4}) i l d e O ( d 3/4 H 2 K 3/4 ) regret bound. We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved O ( H 3 / 2 d K ) O(H^{3/2}d\sqrt{K}) O ( H 3/2 d K ) regret bound for Exo-MDPs, matching lower bounds. New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
New RL approach tackles non-linear MDPs without linear assumptions.
problem Sample efficiency in RL for complex, nonlinear MDPs with continuous states.
method Introduces EPW condition to relax linear structure requirements; provides sample-efficient RL algorithm.
result EPW condition allows solving MDPs without linear assumptions, including Atari games.
Policy gradient methods achieve linear convergence in simple MDPs.
problem Analyzing convergence rates of policy gradient methods in finite MDPs.
method Connections with policy iteration to show linear convergence with large step-sizes.
result Policy gradient methods succeed with large step-sizes and achieve linear rate of convergence.
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
Logarithmic regret achieved in RL with linear function approximation.
problem Achieving logarithmic regret in reinforcement learning with linear function approximation.
method LSVI-UCB for linear MDP assumption, UCRL-VTR for linear mixture MDP assumption.
result Logarithmic regret bounds established for RL with linear function approximation.
Reward-free RL in linear MDPs is as hard as reward-aware RL.
problem Reward-free RL in linear MDPs without access to the reward function during exploration.
method Developed a computationally efficient algorithm with sample complexity O ~ ( d 2 H 5 / ε 2 ) \widetilde{\mathcal{O}}(d^2 H^5/ε^2) O ( d 2 H 5 / ε 2 ) . result Achieved optimal d d d dependence in linear MDPs for reward-free RL, matching the reward-aware RL setting. Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Efficient RL for linear MDPs with unknown transitions.
problem Long planning horizons and unknown state transitions in linear mixture MDPs.
method Horizon-free algorithm using weighted least squares with variance and uncertainty awareness.
result Achieves optimal regret up to logarithmic factors.
New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
Paper optimizes learning MDPs with adversarial rewards using linear models.
problem Learning MDPs with adversarial rewards and changing reward functions.
method Proposes POWERS, an optimistic policy optimization algorithm.
result Achieves nearly optimal regret bound of O ( d H T ) O(dH\sqrt{T}) O ( d H T ) . Improved RL algorithm with linear MDPs for offline learning with partial data coverage.
problem Efficient offline RL with linear MDPs under partial data coverage.
method Primal-dual algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity. result First computationally efficient algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity for offline RL with linear MDPs under partial data coverage. DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
UCRL2-VTR achieves nearly optimal regret for learning MDPs with linear function approximation.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method UCRL2-VTR algorithm with Bernstein-type bonus.
result Achieves a regret of i l d e O ( d D T ) ilde{O}(d\sqrt{DT}) i l d e O ( d D T ) with matching lower bound. Paper proposes a privacy-preserving RL algorithm for linear MDPs with theoretical guarantees.
problem Protecting users' private data in personalized services using RL.
method Local differential privacy (LDP) for RL with linear function approximation.
result Achieves a regret bound of $O(d^{5/4}H^{7/4}T^{3/4}\left(\log(1/δ)
ight)^{1/4}\sqrt{1/\varepsilon})$ for linear mixture MDPs.
New RL algorithm tackles nonstationary MDPs with linear approximations and varying rewards.
problem Nonstationary reinforcement learning with evolving reward and state transition functions.
method Developed a new algorithm LSVI-UCB-Restart with periodic restart, and parameter-free Ada-LSVI-UCB-Restart for unknown variation budgets.
result First minimax dynamic regret lower bound for nonstationary linear MDPs and linear MDPs lower bound.
New algorithm learns sparse linear MDPs with polynomial interactions, improving sample complexity.
problem Learning optimal policies in sparse linear MDPs with limited interactions and unknown features.
method Developed a polynomial-time algorithm using feature selection and emulator for sparse linear MDPs.
result First polynomial-time algorithm for learning near-optimal policies in k-sparse linear MDPs.
Extends linear MDP to handle nonlinear rewards.
problem Restrictive linear MDP assumption limits real-world applicability.
method Proposes Generalized Linear MDP (GLMDP) with GLMs for rewards.
result Develops offline RL algorithms achieving suboptimality guarantees.
Trajectory data suffices for efficient RL in linear MDPs.
problem Efficient offline RL in linear MDPs without state space scaling.
method Linear MDP approximation and trajectory data.
result Trajectory data suffices for deriving ε-optimal policies.
New algorithm reduces switching costs in RL beyond linear MDPs.
problem Costly policy switching in reinforcement learning.
method ELEANOR-LowSwitching algorithm for linear Bellman-complete MDPs.
result Achieves near-optimal regret with logarithmic switching cost.
New algorithm reduces reinforcement learning regret for linear MDPs with unknown transitions.
problem Adversarial linear mixture MDPs with bandit feedback and unknown transition.
method Proposes a new algorithm with a least square estimator and self-normalized concentration.
result Achieves improved regret bound with high probability.
New RL algorithm achieves nearly optimal performance for linear MDPs.
problem Optimal reinforcement learning for episodic linear MDPs.
method Weighted linear regression with variance estimator and rare-switching policy.
result Achieves nearly minimax optimal regret i l d e O ( d H 3 K ) ilde O(d\sqrt{H^3K}) i l d e O ( d H 3 K ) . In this paper we derive an efficient method for computing the indices associated with an asymptotically optimal upper confidence bound algorithm (MDP-UCB) of Burnetas and Katehakis (1997) that only requires solving a system of two non-linear equations with two unknowns, irrespective of the cardinality of the state spac…
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
While designing the state space of an MDP, it is common to include states that are transient or not reachable by any policy (e.g., in mountain car, the product space of speed and position contains configurations that are not physically reachable). This leads to defining weakly-communicating or multi-chain MDPs. In this…
Algorithm for online learning in MDPs with linear function approximation and bandit feedback.
problem Online learning in MDPs with changing reward functions and limited feedback.
method Developed MDP-LinExp3 algorithm with theoretical guarantees.
result Proved regret bounds for MDP-LinExp3 algorithm.
We prove a lower bound for feature dimension in linear MDPs and propose a novel dynamics aggregation framework.
problem The limitation of feature dimension in linear MDPs and the need for efficient hierarchical reinforcement learning.
method We propose a novel dynamics aggregation framework based on structural dynamics and design a provably efficient hierarchical reinforcement learning algorithm.
result Our algorithm achieves a regret of i l d e O ( d ψ 3 / 2 H 3 / 2 N T ) ilde{O} ( d_ψ^{3/2} H^{3/2}\sqrt{ N T} ) i l d e O ( d ψ 3/2 H 3/2 N T ) and meets the condition d ψ 3 N ≪ d 3 d_ψ^3 N \ll d^{3} d ψ 3 N ≪ d 3 in most real-world environments. New algorithms identify best policies in discounted linear MDPs efficiently.
problem Identifying the best policy in discounted linear MDPs with limited samples.
method Derive lower bounds and devise simple yet near-optimal algorithms.
result Upper bound on sample complexity matches existing bounds.
Improved exploration in adversarial MDPs via dilated bonuses.
problem Global exploration in adversarial MDPs with local-search policies.
method Dilated bonuses to facilitate global exploration.
result Improved regret bounds in various MDP settings.
New model selects robustly in adversarial reinforcement learning with unknown corruption.
problem Adversarial corruption in reinforcement learning with unknown total corruption amount.
method Model selection approach for finite-horizon tabular and linear MDPs.
result First worst-case optimal bound without knowledge of total corruption.
Optimistic actor-critic tackles linear MDPs with parametric policies.
problem Theoretical limitations of existing actor-critic methods for linear MDPs.
method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.
result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
New method for optimistic planning in MDPs using regularization.
problem Optimistic planning in infinite-horizon discounted MDPs.
method Regularized dynamic programming for approximate value iteration.
result Achieves near-optimal statistical guarantees in learning policies.
New RL algorithm for linear MDPs with nearly optimal regret.
problem Optimizing reinforcement learning for linear mixture Markov decision processes.
method Proposed a new Bernstein-type concentration inequality for self-normalized martingales and a computationally efficient algorithm UCRL-VTR+.
result UCRL-VTR+ achieves nearly minimax optimal regret of i l d e O ( d H T ) ilde O(dH\sqrt{T}) i l d e O ( d H T ) . We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where ba…
Improved regret bound for adversarial MDPs with linear function approximation.
problem Learning in adversarial MDPs with changing loss functions and large state spaces.
method Two algorithms: refined FTRL with log-barrier regularizer and magnitude-reduced loss estimator.
result Achieved i l d e O ( K ) ilde{\mathcal O}(\sqrt K) i l d e O ( K ) regret, improving over i l d e O ( K 2 / 3 ) ilde{\mathcal O}(K^{2/3}) i l d e O ( K 2/3 ) . Optimistic algorithm reduces regret in non-stationary linear MDPs.
problem Efficient learning in non-stationary linear MDPs with evolving reward and transition.
method OPT-WLSVI, an optimistic model-free algorithm using exponential weights.
result Achieves a regret bound of O ~ ( d 5 / 4 H 2 Δ 1 / 4 K 3 / 4 ) \widetilde{\mathcal{O}}(d^{5/4}H^2 Δ^{1/4} K^{3/4}) O ( d 5/4 H 2 Δ 1/4 K 3/4 ) . PROPO tackles non-stationary MDPs with efficient policy optimization.
problem Non-stationary MDPs with varying reward and transition kernels.
method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.
result PROPO achieves near-optimal performance in non-stationary MDPs.
New algorithms for learning MDPs with linear approximations in infinite-horizon settings.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method Optimism principle, adversarial linear bandits, Natural Policy Gradient.
result Efficient algorithms with optimal or near-optimal regret bounds.
Paper improves sample complexity for reward-free RL in low-rank MDPs.
problem Reward-free RL in low-rank MDPs with unknown representation and weights.
method Proposes a novel model-based algorithm RAFFLE with improved sample complexity.
result RAFFLE achieves ε ε ε -optimal policy and accurate system identification with significantly fewer samples. New algorithm learns policies without explicit rewards for MDPs.
problem Learning policies for MDPs without explicit rewards.
method UCRL-RFE algorithm with linear function approximation.
result UCRL-RFE achieves optimal policy with minimal exploration samples.
Paper presents an efficient algorithm for linear MDP with low switching cost.
problem Large state space reinforcement learning problems with low switching cost.
method First algorithm for linear MDP with low switching cost, achieving near-optimal regret and switching cost.
result Regret bound of $\widetilde{O}\left(\sqrt{d^3H^4K}
ight)$ and near-optimal switching cost of $O\left(d H\log K
ight)$ .
Significant improvements in regret analysis for adaptive online learning problems.
problem Exploiting low variance in online learning problems without known variances.
method Novel peeling-based regret analysis leveraging elliptical potential `count` lemma.
result Significant improvements in regret bounds for linear bandits and linear mixture MDPs.
Algorithm learns from offline data to improve performance in target environment.
problem Learning from offline data in a target environment with unknown shifts.
method Adaptive algorithm that uses offline data to improve performance when informative.
result Algorithm provably improves performance over purely online learning when offline data are informative.
New method achieves nearly horizon-free offline reinforcement learning for tabular and linear MDPs.
problem Achieving efficient reinforcement learning without knowing the horizon.
method Simple recursion-based method to bound total variance in offline scenarios.
result Nearly horizon-free sample complexity bounds for offline policy evaluation and optimization.