We characterize value functions in partially observable MDPs as semi-algebraic sets.
problem Understanding feasible value functions in partially observable Markov decision processes.
method Characterization of feasible value functions as semi-algebraic sets defined by polynomial inequalities.
result The feasible set of value functions in POMDPs is a semi-algebraic set, not a polytope as in MDPs.
PRL improves off-policy evaluation in partially observed MDPs.
problem Confounding and bias in offline reinforcement learning with unobserved state factors.
method Extends proximal causal inference to POMDPs, identifying and estimating target policy value.
result Semiparametrically efficient estimators for PRL in partially observed MDPs.
Paper analyzes history-based RL methods for MDPs, introduces a theoretical framework and practical algorithm.
problem Improving RL performance in MDPs using history-based features.
method Theoretical framework for history-based RL, practical algorithm design.
result Practical RL algorithm shows effectiveness on continuous control tasks.
RL approach for target tracking with unknown dynamics and sensor control.
problem Tracking an unknown target with sensor control.
method Track-MDP formulation for RL, compared with POMDP.
result Optimal RL policy tracks all target paths with certainty.
New framework for reinforcement learning with sporadic state observations.
problem Partial observability in reinforcement learning.
method Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs).
result Optimistic algorithm achieving regret bound for episodic learning.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
Unified framework for efficient RL in MDP, POMDP, and PSR.
problem Sample efficient reinforcement learning under various decision-making frameworks.
method Proposed GEC complexity measure and a posterior sampling algorithm.
result Sample efficient learning achieved with sublinear regret bound in terms of GEC.
OMLE combines optimism and MLE for efficient sequential decision making.
problem Efficiently solving sequential decision making problems, especially in partially observable settings.
method Combines optimism for exploration and maximum likelihood estimation for model learning.
result OMLE learns near-optimal policies for a wide range of sequential decision making problems.
Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There is considerable interest in designing reinforcement learning (RL) algorithms with similar properties. This includes proposals to learn the learning algorithm itself, an idea also known as meta learnin…
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
RL struggles with generalization due to implicit partial observability.
problem Generalization in RL is difficult due to implicit partial observability.
method Re-cast RL problem as solving epistemic POMDPs and propose ensemble-based techniques.
result Simple ensemble-based technique achieves significant generalization gains.
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
New algorithm learns optimal policies in strategic MDPs with private types.
problem Optimal policy learning in strategic MDPs with private types and information asymmetry.
method PLAN algorithm using instrumental variable regression and pessimism principle.
result PLAN achieves near-optimal policy with 1 / K 1 / \sqrt{K} 1/ K optimality. Deep active inference learns policies from sensory inputs.
problem Learning policies in partially observable domains.
method Optimizes expected free energy with a variational autoencoder.
result Comparable or better performance than deep Q-learning.
Improved RL algorithm with linear MDPs for offline learning with partial data coverage.
problem Efficient offline RL with linear MDPs under partial data coverage.
method Primal-dual algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity. result First computationally efficient algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity for offline RL with linear MDPs under partial data coverage. Study on future-dependent value functions for off-policy evaluation in complex environments.
problem Exponential dependence on horizon in off-policy evaluation for complex observations.
method Developed novel coverage assumptions for POMDPs to achieve polynomial bounds.
result Achieved polynomial bounds on previously exponential quantities, improving off-policy evaluation.
Algorithm improves reinforcement learning in MDPs with partial order policies.
problem Improving reinforcement learning in MDPs with partial order policies.
method Epoch-based reinforcement learning algorithm leveraging a partial order over policy class.
result Achieves an O ( w log ( ∣ Θ ∣ ) T ) O(\sqrt{w \log(|Θ|) T}) O ( w log ( ∣Θ∣ ) T ) regret bound, independent of state and action space sizes. New algorithm tackles confounding in offline RL for partially observable MDPs.
problem Confounding in offline reinforcement learning for partially observable MDPs.
method P3O algorithm using proximal causal inference and pessimistic confidence regions.
result Achieves n − 1 / 2 n^{-1/2} n − 1/2 -suboptimality for offline RL in POMDPs with confounded data. CPPO learns policies from partial offline data in MDPs with structural assumptions.
problem Offline Reinforcement Learning with partial coverage assumption.
method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.
result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
Transformer learns to infer partial MDPs for efficient in-context adaptation and exploration.
problem Efficiently adapt and explore in-context without gradient-based updates.
method Uses a transformer to learn inference from training tasks, considering hypothesis space of partial models.
result Adaptation speed and exploration-exploitation balance approach those of an exact posterior sampling oracle.
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an…
Reinforcement Learning (RL) is a learning paradigm concerned with learning to control a system so as to maximize an objective over the long term. This approach to learning has received immense interest in recent times and success manifests itself in the form of human-level performance on games like \textit{Go}. While R…
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to de…
We solve a broad class of sequential decision-making problems with partially observed states.
problem Sequential decision-making under uncertainty with partially observed states.
method Modeling as a partially observed Markov decision process (POMDP) and separating state and modulation process.
result The approach allows for specialized approximate solution procedures.
New UCB algorithm for learning PSRs with tractable computation and accuracy.
problem Learning predictive state representations in sequential decision-making problems.
method Proposes a novel UCB-type algorithm with a bonus term to estimate PSRs accurately and efficiently.
result First known UCB-type approach for PSRs with guaranteed model accuracy and computational tractability.
DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
New method for off-policy evaluation in POMDPs using future-dependent value functions.
problem Curse of horizon in off-policy evaluation for POMDPs.
method Develops future-dependent value functions and minimax learning method.
result PAC result and Bellman completeness for the proposed OPE estimator.
Study on natural actor-critic for POMDPs with finite memory.
problem Learning in partially observed Markov decision processes with noisy observations.
method Finite actor-critic method with multi-step temporal difference learning.
result First non-asymptotic global convergence for POMDPs with function approximation.
Optimal strategy for reinforcement learning with delayed observations.
problem Delayed state observation in reinforcement learning.
method Combines augmentation method and upper confidence bound approach.
result Minimax optimal regret bound of i l d e O ( H D max S A K ) ilde{\mathcal{O}}(H \sqrt{D_{\max} SAK}) i l d e O ( H D m a x S A K ) . Reinforcement learning (RL) in Markov decision processes (MDPs) with large state spaces is a challenging problem. The performance of standard RL algorithms degrades drastically with the dimensionality of state space. However, in practice, these large MDPs typically incorporate a latent or hidden low-dimensional structu…
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
We decode latent states in Block MDPs and learn near-optimal policies.
problem Model estimation and reward-free learning in Block MDPs.
method Information-theoretical lower bound and efficient model estimation algorithm.
result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.
This paper establishes strong lower bounds for learning in revealing POMDPs.
problem Understanding the fundamental limits of reinforcement learning in revealing partially observable Markov Decision Processes (POMDPs).
method Develops strong PAC and regret lower bounds for learning in revealing POMDPs using multi-step revealing POMDPs as a case study.
result Strong polynomial lower bounds for learning in revealing POMDPs, achieving significantly smaller gaps against current upper bounds.
In order to make good decision under uncertainty an agent must learn from observations. To do so, two of the most common frameworks are Contextual Bandits and Markov Decision Processes (MDPs). In this paper, we study whether there exist algorithms for the more general framework (MDP) which automatically provide the bes…
This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.
problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.
method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.
result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Hypothesis testing is an important problem with applications in target localization, clinical trials etc. Many active hypothesis testing strategies operate in two phases: an exploration phase and a verification phase. In the exploration phase, selection of experiments is such that a moderate level of confidence on the …
Proposes method to learn state abstractions that generalize across environments.
problem Learning abstractions that generalize in block MDPs.
method Invariant causal prediction to learn model-irrelevant state abstractions (MISA).
result Proves high probability of outputting a state abstraction corresponding to causal feature set for return.
New framework for reinforcement learning with adversarial preferences in tabular MDPs.
problem Learning from preferences rather than direct rewards in MDPs with adversarial settings.
method Developed PbMDPs framework, established lower bounds, and proposed algorithms for regret minimization.
result Achieved regret bounds of Ω ( ( H 2 S K ) 1 / 3 T 2 / 3 ) Ω( (H^2 S K)^{1/3} T^{2/3} ) Ω (( H 2 S K ) 1/3 T 2/3 ) for PbMDPs with Borda scores. Study shows offline RL with partial coverage and weak function classes is possible.
problem Learning optimal policies from logged data with function approximation.
method Marginalized Importance Sampling (MIS) with additional covering distribution.
result Finite-sample guarantees for sample-efficient offline RL for general MDPs.
A new method uses a frozen language model to improve sample efficiency in reinforcement learning.
problem Improving sample efficiency in reinforcement learning with partially observable environments.
method FROZEN Hopfield network and HELM (History Embedding Language Model) method.
result HELM achieves new state-of-the-art results on Minigrid and Procgen environments.
New k k k -step policy gradient method avoids local optima in restricted policy classes.
problem Suboptimal local optima in policy gradient methods for restricted policy classes.
method Proposes a k k k -step policy gradient method to escape myopic local optima. result The method converges to near optimal solutions exponentially close to the optimal deterministic policy.
Survey of integrating planning and learning in model-based reinforcement learning.
problem Sequential decision making in AI, formalized as MDP optimization.
method Systematic coverage of dynamics model learning and planning-learning integration.
result Broad conceptual overview of model-based reinforcement learning.
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular …
This paper investigates multi-task reinforcement learning in non-Markovian decision making, showing benefits in sample efficiency.
problem Investigating multi-task reinforcement learning in non-Markovian decision making processes.
method Developed a joint model class for tasks and used the η η η -bracketing number to quantify complexity and similarity. result Multi-task reinforcement learning can improve sample efficiency in non-Markovian decision making processes.
The paper extends conformal prediction to MDP trajectories for autonomous systems.
problem Ensuring reliability of autonomous systems by providing probabilistic guarantees.
method Applying conformal corrections to quantile regression prediction intervals.
result Conformal prediction intervals ensure the observed trajectory lies inside with high probability.
This work tackles representation learning for RL in low-rank MDPs, improving sample efficiency.
problem How to learn a compact representation for RL in low-rank MDPs efficiently.
method Proposes REP-UCB for online RL and develops an algorithm for offline RL under partial coverage.
result Significantly improved sample complexity for online RL and competitive performance for offline RL.