Sparse MDP with entropy regularization improves reinforcement learning performance.
problem Improving reinforcement learning policies with sparse and multi-modal distributions.
method Proposes a sparse Markov decision process with causal sparse Tsallis entropy regularization.
result The proposed method achieves a constant performance error bound, outperforming soft MDPs.
This paper introduces a fast algorithm for solving MDPs with sparse rewards.
problem Solving MDPs with large state and action spaces and sparse reward sources is computationally expensive.
method A novel algorithm that solves deterministic, continuous MDPs with sparse reward sources efficiently and exactly.
result The algorithm offers a time complexity of O ( ∣ R ∣ 2 i m e s ∣ A ∣ 2 i m e s ∣ S ∣ ) O( |R|^2 imes |A|^2 imes |S|) O ( ∣ R ∣ 2 im es ∣ A ∣ 2 im es ∣ S ∣ ) and a memory complexity of O ( ∣ S ∣ + ∣ R ∣ i m e s ∣ A ∣ ) O( |S| + |R| imes |A|) O ( ∣ S ∣ + ∣ R ∣ im es ∣ A ∣ ) . Proposes a framework for sparse optimal policies in reinforcement learning.
problem Finding sparse optimal policies in reinforcement learning.
method Regularized Markov decision processes (MDPs) with specific regularization terms.
result Sufficient and necessary conditions for inducing sparse optimal policies.
Exact solution for sparse-reward MDPs with minimal state space dependence.
problem Finding optimal policies for MDPs with sparse rewards and large state spaces.
method Proposes an algorithm with time complexity O ( ∣ R ∣ 3 i m e s ∣ A ∣ 2 ) O( |R|^3 imes |A|^2 ) O ( ∣ R ∣ 3 im es ∣ A ∣ 2 ) and memory complexity O ( ∣ R ∣ i m e s ∣ A ∣ ) O( |R| imes |A| ) O ( ∣ R ∣ im es ∣ A ∣ ) for exact computation. result Exact policy computation without state space dependency for sparse-reward MDPs.
New algorithm learns sparse linear MDPs with polynomial interactions, improving sample complexity.
problem Learning optimal policies in sparse linear MDPs with limited interactions and unknown features.
method Developed a polynomial-time algorithm using feature selection and emulator for sparse linear MDPs.
result First polynomial-time algorithm for learning near-optimal policies in k-sparse linear MDPs.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
Study shows high-dimensional sparse RL hardness and Lasso Q-iteration's nearly dimension-free regret.
problem Hardness of online sparse reinforcement learning in high-dimensional MDPs.
method Lower bound construction and Lasso fitted Q-iteration analysis.
result Lasso Q-iteration achieves nearly dimension-free regret of O ~ ( s 2 / 3 N 2 / 3 ) \tilde{O}(s^{2/3}N^{2/3}) O ~ ( s 2/3 N 2/3 ) with oracle access to a good exploratory policy. New concept of Blackwell regret for reinforcement learning with sparse rewards.
problem Sparse rewards in long horizon MDPs.
method Formalization of myopic discount factors, value functions, and policies in terms of Blackwell optimality; introduction of Blackwell regret.
result Selecting a discount factor for zero Blackwell regret becomes arbitrarily hard in long horizon MDPs.
VO Q Q Q L optimizes RL with sparse rewards using weighted bounds.
problem Sparse rewards and non-linear function approximation in RL.
method VO Q Q Q L combines Q Q Q -learning with weighted bounds for optimal regret. result Achieves asymptotically optimal regret for linear function approximation.
Sparse PCL algorithms improve optimal policy in Tsallis entropy-regularized MDPs.
problem Sparse optimal policies in Tsallis entropy-regularized MDPs.
method Path consistency learning (PCL) algorithms for sparse entropy-regularized RL.
result Sparse PCL algorithms reduce sub-optimality compared to soft ERL, especially in high-action problems.
UCB-TQL learns from multiple tasks with shared dynamics and adapts to task-specific variations.
problem Transfer reinforcement learning with composite MDPs where tasks share core dynamics but have sparse differences.
method UCB-TQL, a novel transfer RL algorithm for composite MDPs.
result Achieved a regret bound of i l d e O ( e H 5 N ) ilde{O}(\sqrt{eH^5N}) i l d e O ( e H 5 N ) that scales independently of the ambient dimension. A reinforcement learning framework for Mars rover control using temporal logic.
problem Sparse rewards in continuous-state continuous-action MDPs with high-level temporal structures.
method Actor-critic, model-free, online RL framework with modular DDPG architecture.
result Success rate of synthesised policy in Mars rover experiment.
New algorithm reduces switching costs in RL beyond linear MDPs.
problem Costly policy switching in reinforcement learning.
method ELEANOR-LowSwitching algorithm for linear Bellman-complete MDPs.
result Achieves near-optimal regret with logarithmic switching cost.
A new method for estimating joint value functions in multi-scene reinforcement learning.
problem High variance in samples for policy gradient computations in multi-scene environments.
method Sparse attention mechanism over multiple value function hypotheses to approximate the true joint value function.
result Significant improvements in reward scores and enhanced navigation efficiency across OpenAI ProcGen environments.
Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.
problem Exploration-exploitation dilemma in finite-horizon reinforcement learning with metric state-action spaces.
method Kernel-UCBVI, leveraging smoothness and kernel estimators of rewards and transitions.
result First regret bound for kernel-based RL using smoothing kernels, O ( H 3 K 2 d / ( 2 d + 1 ) ) O(H^3 K^{2d/(2d+1)}) O ( H 3 K 2 d / ( 2 d + 1 ) ) . Paper improves reinforcement learning in multi-scene tasks.
problem Reducing sample variance in multi-scene reinforcement learning.
method Sparse dynamic value estimation using Gaussian mixture models.
result Significant improvements in reward scores and navigation efficiency.
ETGL-DDPG improves DDPG for sparse reward control with new exploration and replay techniques.
problem Sparse reward continuous control in reinforcement learning.
method Introduces ε t εt εt -greedy search and GDRB framework for efficient exploration and reward use. result ETGL-DDPG outperforms DDPG and other methods on sparse-reward continuous benchmarks.
Off-policy evaluation for MNAR rewards in MDPs
problem Off-policy evaluation in MDPs with MNAR rewards
method Formalizing a reward-dependent propensity model and using future states as shadow variables
result Proposed an Fitted-Q-Evaluation-style estimator that propagates recovered rewards while allowing target policies to depend on past missingness indicators
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
Improved exploration methods for reinforcement learning with reduced sample complexity.
problem Challenges in reinforcement learning exploration in unknown environments.
method Proposed game-theoretic and trajectory entropy algorithms with improved sample complexity.
result Established statistical advantage of entropy-regularized MDPs for exploration and reduced sample complexity.
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved O ( H 3 / 2 d K ) O(H^{3/2}d\sqrt{K}) O ( H 3/2 d K ) regret bound for Exo-MDPs, matching lower bounds. Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. ARL algorithm reduces adversarial MDP to bandit problems for reliable policy learning.
problem Learning reliable policies in non-stationary, adversarial MDPs.
method Adversarial Reinforcement Learning (ARL) algorithm that converts MDP to a sequence of adversarial bandit problems.
result Achieves optimal regret bound of O ( S A T H 3 ) O(\sqrt{SATH^3}) O ( S A T H 3 ) . Deep RL solves combinatorial selection problems with large item spaces.
problem Solving MDPs with large state and action spaces, especially for combinatorial selection.
method Convert S-MDP to IS-MDP, use weight-shared Q-networks to manage state space explosion.
result Our approach effectively handles large item spaces and scales to diverse environments.
Meta RL learns task structure from experience.
problem Designing efficient reinforcement learning algorithms.
method Separately learns policy and task belief using privileged information.
result Effective at solving complex meta-RL environments.
We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
A new method for CMDP solving without compromising safety constraints.
problem Solving CMDP problems while adhering to safety constraints.
method Decomposition into reconnaissance and planning MDPs.
result Achieves safe policies for any safety constraint set.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
Optimistic algorithms achieve logarithmic regret bounds for MDPs without diameter dependence.
problem Achieving logarithmic regret bounds for episodic MDPs without relying on diameter-like quantities.
method Novel 'clipped' regret decomposition applied to optimistic algorithms.
result Smooth interpolation between gap-dependent and minimax rates of convergence.
Paper proposes efficient sample collection strategy for RL.
problem Balancing exploration and exploitation in reinforcement learning.
method Decoupled approach with objective-specific and objective-agnostic strategies.
result Improved or novel sample complexity guarantees for various RL settings.
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Reward suffices for convex MDPs, expanding RL to new problems.
problem Capturing goals as convex functions of stationary distribution.
method Reformulated as a min-max game using Fenchel duality.
result Convex MDPs require non-stationary reward functions.
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
Study on regret minimization in deterministic MDPs.
problem Minimizing regret in deterministic reinforcement learning.
method Logarithmic regret lower bounds, leveraging graph theory and cycles.
result Explicitly quantifies the fundamental limit of performance achievable by any learning algorithm.
Improved regret bound for MNL MDPs with variance-aware approach.
problem Optimal reinforcement learning for MNL MDPs with structured variance.
method Introducing a problem-dependent constant measuring average variance, proposing an algorithm with improved regret bound.
result Minimax optimal regret bound of O ( d H 2 σ ˉ T T ) O(dH^2\barσ_T\sqrt{T}) O ( d H 2 σ ˉ T T ) for structured MDPs. TUCRL efficiently explores and exploits in non-communicating MDPs without prior knowledge.
problem Efficient exploration-exploitation in non-communicating Markov Decision Processes (MDPs).
method Introduces TUCRL, the first algorithm for efficient exploration-exploitation in any finite MDP without prior knowledge.
result Derives a O ~ ( D e x t t t C Γ e x t t t C S e x t t t C A T ) \widetilde{O}(D^{ exttt{C}} \sqrt{Γ^{ exttt{C}} S^{ exttt{C}} AT}) O ( D e x ttt C Γ e x ttt C S e x ttt C A T ) regret bound for weakly-communicating MDPs. This work uses action equivariance to learn structured latent spaces for reinforcement learning.
problem Learning structured latent spaces for reinforcement learning.
method Introduced a contrastive loss function to enforce action equivariance on learned representations.
result Optimal policies in the abstract MDP can be successfully lifted to the original MDP.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
New algorithms improve exploration in MDPs with theoretical guarantees.
problem Efficient exploration in undiscounted MDPs with continuous states.
method Exploration bonuses for SCAL and C-SCAL algorithms.
result Achieves sublinear regret with improved computational efficiency.
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
The paper introduces MDP homomorphic networks for faster reinforcement learning.
problem Current reinforcement learning approaches do not exploit symmetries in the joint state-action space.
method Equivariant neural networks with group-structured symmetries (reflections, rotations).
result MDP homomorphic networks converge faster than unstructured baselines on various tasks.
UCBVI-γ algorithm minimizes regret in discounted MDPs.
problem Minimizing regret in discounted MDPs.
method Optimism in the face of uncertainty principle and Bernstein-type bonus.
result UCBVI-γ achieves nearly minimax optimal regret.