Spectral method learns hidden state mapping for RL in rich-observation MDPs.
problem Challenges in RL with large state spaces and hidden low-dimensional structure.
method Spectral decomposition method to learn hidden state to observation state mapping.
result Achieves low regret with weak dependence on observed space dimensionality.
Method learns latent states from rich observations to improve RL exploration.
problem Improving RL performance with rich observations and latent states.
method Estimates latent states from observations through regression and clustering, providing finite-sample guarantees.
result Exponential improvement over Q Q Q -learning with naïve exploration. This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.
problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.
method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.
result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.
We decode latent states in Block MDPs and learn near-optimal policies.
problem Model estimation and reward-free learning in Block MDPs.
method Information-theoretical lower bound and efficient model estimation algorithm.
result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.
New algorithms for efficient reinforcement learning with rich observations.
problem Efficient reinforcement learning with rich observations in environments with deterministic dynamics.
method Oracle-efficient algorithms using standard optimization primitives.
result Proved sample efficiency and presented examples of challenges.
New RL algorithms show model-based methods are more efficient than model-free ones in complex decision processes.
problem Efficient reinforcement learning in contextual decision processes with strategic exploration.
method Design of new model-based RL algorithms with sample complexity governed by witness rank.
result Exponential separation between model-based and model-free RL in some rich-observation settings.
Paper tackles reinforcement learning with complex observations and simple latent dynamics.
problem Understanding reinforcement learning with complex observations and simple latent dynamics.
method Statistical and algorithmic analysis of reinforcement learning under general latent dynamics.
result Identifies latent pushforward coverability as a condition for statistical tractability.
The paper introduces a method to learn Markov state abstractions for reinforcement learning.
problem Learning Markov state representations in complex environments.
method The paper introduces a novel set of conditions and a training procedure combining inverse model estimation and temporal contrastive learning.
result The approach learns representations that capture the underlying structure of the domain and improve sample efficiency.
Efficiently extracts linear dynamics from complex observations.
problem Learning policies directly from rich, high-dimensional observations.
method Modeling linear dynamics in a hidden subspace and developing an efficient algorithm.
result Successfully extracts linear dynamics from rich observations.
Method learns representations invariant to task-irrelevant details in reinforcement learning tasks.
problem Learning representations that are invariant to task-irrelevant details in reinforcement learning.
method Uses bisimulation metrics to learn robust latent representations that encode only task-relevant information.
result Demonstrates SOTA performance in modified visual MuJoCo tasks and a first-person driving task.
We propose and study a new model for reinforcement learning with rich observations, generalizing contextual bandits to sequential decision making. These models require an agent to take actions based on observations (features) with the goal of achieving long-term performance competitive with a large set of policies. To …
HOMER learns latent states to explore rich environments efficiently.
problem Exploration in rich observation environments with unknown latent states.
method Interleaves representation learning and strategic exploration to identify kinematic states.
result Provably efficient exploration with polynomial sample complexity in latent states and time horizon.
The paper shows how to learn near-optimal behavior in reinforcement learning with rich observations.
problem Learning near-optimal behavior in reinforcement learning with rich observations and function approximation.
method Introduces a new model called contextual decision processes and a new algorithm that engages in systematic exploration to learn these processes with low Bellman rank.
result The algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters.
Sparse MDP with entropy regularization improves reinforcement learning performance.
problem Improving reinforcement learning policies with sparse and multi-modal distributions.
method Proposes a sparse Markov decision process with causal sparse Tsallis entropy regularization.
result The proposed method achieves a constant performance error bound, outperforming soft MDPs.
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved O ( H 3 / 2 d K ) O(H^{3/2}d\sqrt{K}) O ( H 3/2 d K ) regret bound for Exo-MDPs, matching lower bounds. Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. ARL algorithm reduces adversarial MDP to bandit problems for reliable policy learning.
problem Learning reliable policies in non-stationary, adversarial MDPs.
method Adversarial Reinforcement Learning (ARL) algorithm that converts MDP to a sequence of adversarial bandit problems.
result Achieves optimal regret bound of O ( S A T H 3 ) O(\sqrt{SATH^3}) O ( S A T H 3 ) . Deep RL solves combinatorial selection problems with large item spaces.
problem Solving MDPs with large state and action spaces, especially for combinatorial selection.
method Convert S-MDP to IS-MDP, use weight-shared Q-networks to manage state space explosion.
result Our approach effectively handles large item spaces and scales to diverse environments.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
A new method for CMDP solving without compromising safety constraints.
problem Solving CMDP problems while adhering to safety constraints.
method Decomposition into reconnaissance and planning MDPs.
result Achieves safe policies for any safety constraint set.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
Optimistic algorithms achieve logarithmic regret bounds for MDPs without diameter dependence.
problem Achieving logarithmic regret bounds for episodic MDPs without relying on diameter-like quantities.
method Novel 'clipped' regret decomposition applied to optimistic algorithms.
result Smooth interpolation between gap-dependent and minimax rates of convergence.
Study reduces real-world sample complexity for reinforcement learning.
problem Decrease real-world sample complexity in reinforcement learning.
method Formulates a theoretical framework for Sim-to-Real reinforcement learning using ROMDPs.
result Establishes real-world sample complexity guarantees smaller than direct learning.
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Reward suffices for convex MDPs, expanding RL to new problems.
problem Capturing goals as convex functions of stationary distribution.
method Reformulated as a min-max game using Fenchel duality.
result Convex MDPs require non-stationary reward functions.
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
Study on regret minimization in deterministic MDPs.
problem Minimizing regret in deterministic reinforcement learning.
method Logarithmic regret lower bounds, leveraging graph theory and cycles.
result Explicitly quantifies the fundamental limit of performance achievable by any learning algorithm.
Improved regret bound for MNL MDPs with variance-aware approach.
problem Optimal reinforcement learning for MNL MDPs with structured variance.
method Introducing a problem-dependent constant measuring average variance, proposing an algorithm with improved regret bound.
result Minimax optimal regret bound of O ( d H 2 σ ˉ T T ) O(dH^2\barσ_T\sqrt{T}) O ( d H 2 σ ˉ T T ) for structured MDPs. TUCRL efficiently explores and exploits in non-communicating MDPs without prior knowledge.
problem Efficient exploration-exploitation in non-communicating Markov Decision Processes (MDPs).
method Introduces TUCRL, the first algorithm for efficient exploration-exploitation in any finite MDP without prior knowledge.
result Derives a O ~ ( D e x t t t C Γ e x t t t C S e x t t t C A T ) \widetilde{O}(D^{ exttt{C}} \sqrt{Γ^{ exttt{C}} S^{ exttt{C}} AT}) O ( D e x ttt C Γ e x ttt C S e x ttt C A T ) regret bound for weakly-communicating MDPs. This work uses action equivariance to learn structured latent spaces for reinforcement learning.
problem Learning structured latent spaces for reinforcement learning.
method Introduced a contrastive loss function to enforce action equivariance on learned representations.
result Optimal policies in the abstract MDP can be successfully lifted to the original MDP.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
New algorithms improve exploration in MDPs with theoretical guarantees.
problem Efficient exploration in undiscounted MDPs with continuous states.
method Exploration bonuses for SCAL and C-SCAL algorithms.
result Achieves sublinear regret with improved computational efficiency.
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
The paper introduces MDP homomorphic networks for faster reinforcement learning.
problem Current reinforcement learning approaches do not exploit symmetries in the joint state-action space.
method Equivariant neural networks with group-structured symmetries (reflections, rotations).
result MDP homomorphic networks converge faster than unstructured baselines on various tasks.
UCBVI-γ algorithm minimizes regret in discounted MDPs.
problem Minimizing regret in discounted MDPs.
method Optimism in the face of uncertainty principle and Bernstein-type bonus.
result UCBVI-γ achieves nearly minimax optimal regret.
New RL approach tackles non-linear MDPs without linear assumptions.
problem Sample efficiency in RL for complex, nonlinear MDPs with continuous states.
method Introduces EPW condition to relax linear structure requirements; provides sample-efficient RL algorithm.
result EPW condition allows solving MDPs without linear assumptions, including Atari games.
New method for estimating and optimizing MDPs without stationarity.
problem Challenges in offline contextual MDP estimation without stationarity.
method Introduces a new adaptive estimation and cost optimization approach for contextual MDPs.
result First robust, theoretically backed method for offline contextual MDP estimation.
This paper introduces a fast algorithm for solving MDPs with sparse rewards.
problem Solving MDPs with large state and action spaces and sparse reward sources is computationally expensive.
method A novel algorithm that solves deterministic, continuous MDPs with sparse reward sources efficiently and exactly.
result The algorithm offers a time complexity of O ( ∣ R ∣ 2 i m e s ∣ A ∣ 2 i m e s ∣ S ∣ ) O( |R|^2 imes |A|^2 imes |S|) O ( ∣ R ∣ 2 im es ∣ A ∣ 2 im es ∣ S ∣ ) and a memory complexity of O ( ∣ S ∣ + ∣ R ∣ i m e s ∣ A ∣ ) O( |S| + |R| imes |A|) O ( ∣ S ∣ + ∣ R ∣ im es ∣ A ∣ ) . Unified framework for solving MDPs with stochastic mirror descent.
problem Approximately solving infinite-horizon Markov decision processes (MDPs).
method Primal-dual stochastic mirror descent for MDPs with a unified framework.
result Computes ε-optimal policies with expected samples for both average-reward and discounted MDPs.
Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a i l d e O ( d 3 / 4 H 2 K 3 / 4 ) ilde{\mathcal{O}}(d^{3/4}H^2K^{3/4}) i l d e O ( d 3/4 H 2 K 3/4 ) regret bound. ASE safely explores unknown MDPs with unknown dynamics, improving sample efficiency.
problem Balancing exploration and safety in unknown MDPs with stochastic dynamics.
method Exploits analogies between state-action pairs to safely learn near-optimal policies.
result Empirically improves sample efficiency compared to existing methods.