New RL algorithm maximizes CVaR in low-rank MDPs with provable efficiency.
problem Maximizing CVaR in large state spaces with function approximation.
method Upper Confidence Bound (UCB) bonus-driven algorithm for low-rank MDPs.
result Achieves sample complexity of O(H^7 A^2 d^4 / τ^2 ε^2) for ε-optimal CVaR.
FLAMBE tackles RL in low rank MDPs by learning features.
problem Dealing with the curse of dimensionality in RL.
method Develops FLAMBE, a method that engages in exploration and representation learning for RL in low rank transition models.
result FLAMBE efficiently learns features for RL in low rank transition models.
Paper improves sample complexity for reward-free RL in low-rank MDPs.
problem Reward-free RL in low-rank MDPs with unknown representation and weights.
method Proposes a novel model-based algorithm RAFFLE with improved sample complexity.
result RAFFLE achieves ε ε ε -optimal policy and accurate system identification with significantly fewer samples. New model-free algorithms learn representations for low-rank MDPs efficiently.
problem Learning representations in reinforcement learning for low-rank MDPs.
method Developed minimax representation learning objective and interleaved with reward-free exploration.
result Proven sample efficiency and scalability to complex environments.
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
This work tackles representation learning for RL in low-rank MDPs, improving sample efficiency.
problem How to learn a compact representation for RL in low-rank MDPs efficiently.
method Proposes REP-UCB for online RL and develops an algorithm for offline RL under partial coverage.
result Significantly improved sample complexity for online RL and competitive performance for offline RL.
Algorithm POLO learns low-rank MDPs with adversarial changes in full-info feedback.
problem Learning low-rank MDPs with adversarial changes and unknown transition probabilities.
method Policy optimization-based algorithm POLO with regret guarantee.
result POLO achieves sublinear regret guarantee with no dependence on state space size.
New method extends low-rank MDPs to continuous action spaces.
problem Limited applicability of current low-rank MDP methods to continuous action spaces.
method Extending FLAMBE algorithm to continuous action spaces with Hölder smoothness conditions.
result Similar PAC bound achieved for continuous actions with polynomial dependence on smoothness order.
Paper learns meaningful state and action representations from MDP trajectories.
problem Learning good state and action representations from MDP trajectories.
method Tensor decomposition, kernelization, importance sampling, low-Tucker-rank approximation.
result The learned state/action abstractions provide accurate approximations to latent block structures.
This work explores efficient reinforcement learning with density features in low-rank MDPs.
problem Efficient reinforcement learning with density features in low-rank MDPs.
method Proposes algorithms for off-policy estimation and online construction of exploratory data distributions.
result Demonstrates sample-efficient learning with density features in low-rank MDPs, overcoming technical challenges.
Paper presents a reduction-based framework for conservative bandits and RL with improved lower and upper bounds.
problem Conservative bandits and reinforcement learning problems.
method Reduction technique to calculate necessary and sufficient budget from baseline policy.
result Improved lower and upper bounds for various conservative settings.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving ve…
Safe exploration in RF-RL doesn't increase sample complexity.
problem Achieving optimal policies with safety constraints in reward-free RL.
method Proposed SWEET framework for tabular and low-rank MDP settings, leveraging truncated value functions.
result Sample complexities match or outperform constraint-free counterparts, proving safety constraints have little impact.
New RL approach uses resets to learn complex MDPs efficiently.
problem Learning complex MDPs with high-dimensional states and function approximation.
method Local simulator access and recursive value function search.
result Proven sample-efficient learning for MDPs with low coverability.
Study reward-free RL in non-linear settings, improving efficiency and removing assumptions.
problem Improving sample efficiency in reward-free reinforcement learning for non-linear function approximation.
method Proposed RFOLIVE algorithm for minimal structural assumptions, analyzed hardness results for reward-free and reward-aware exploration.
result Statistical efficiency and hardness results under various structural assumptions, no need for reachability or explorability assumptions.
New L 1 L_1 L 1 -Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces L 1 L_1 L 1 -Coverage objective to enable efficient exploration and planning. result First computationally efficient algorithms for online reinforcement learning with low coverability.
UCB-TQL learns from multiple tasks with shared dynamics and adapts to task-specific variations.
problem Transfer reinforcement learning with composite MDPs where tasks share core dynamics but have sparse differences.
method UCB-TQL, a novel transfer RL algorithm for composite MDPs.
result Achieved a regret bound of i l d e O ( e H 5 N ) ilde{O}(\sqrt{eH^5N}) i l d e O ( e H 5 N ) that scales independently of the ambient dimension. Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
New BE dimension measure reveals rich RL problems with sample-efficient algorithms.
problem Finding sample-efficient algorithms for complex RL problems.
method Introducing Bellman Eluder (BE) dimension and designing GOLF and OLIVE algorithms.
result GOLF and OLIVE algorithms learn near-optimal policies for low BE dimension problems with polynomial samples.
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
A new framework reduces RL sample complexity for complex MDPs.
problem Handling large state and action spaces in reinforcement learning.
method Unified model-based and model-free RL framework with ABC class, novel estimation function, and functional eluder dimension.
result OPERA algorithm achieves sample-efficient regret bounds for various MDP models.
Optimistic algorithm reduces regret in non-stationary linear MDPs.
problem Efficient learning in non-stationary linear MDPs with evolving reward and transition.
method OPT-WLSVI, an optimistic model-free algorithm using exponential weights.
result Achieves a regret bound of O ~ ( d 5 / 4 H 2 Δ 1 / 4 K 3 / 4 ) \widetilde{\mathcal{O}}(d^{5/4}H^2 Δ^{1/4} K^{3/4}) O ( d 5/4 H 2 Δ 1/4 K 3/4 ) . New method improves RL in continuous spaces with kernel smoothing.
problem Sample efficiency and structural assumptions in classical RL.
method Kernel smoothing model-based approach with Bernstein-style exploration bonus.
result Achieves improved regret bound in finite-horizon settings.
Unified framework for efficient RL in MDP, POMDP, and PSR.
problem Sample efficient reinforcement learning under various decision-making frameworks.
method Proposed GEC complexity measure and a posterior sampling algorithm.
result Sample efficient learning achieved with sublinear regret bound in terms of GEC.
This work shows how to use simulators to learn efficient exploration in real-world RL.
problem Sample complexity of real-world reinforcement learning.
method Coupling exploratory policies learned in simulators with practical approaches.
result Polynomial sample complexity in real world, exponential improvement over direct sim2real transfer.
OMLE combines optimism and MLE for efficient sequential decision making.
problem Efficiently solving sequential decision making problems, especially in partially observable settings.
method Combines optimism for exploration and maximum likelihood estimation for model learning.
result OMLE learns near-optimal policies for a wide range of sequential decision making problems.
CPPO learns policies from partial offline data in MDPs with structural assumptions.
problem Offline Reinforcement Learning with partial coverage assumption.
method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.
result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Paper tackles robust offline RL for non-Markovian processes, improving efficiency and applicability.
problem Learning robust policies for non-Markovian decision processes with limited offline data.
method Proposes a novel algorithm with dataset distillation and LCB design for robust values, derived new dual forms, and introduces concentrability coefficients.
result Proves polynomial sample efficiency for finding ε-optimal robust policies.
Paper presents an efficient algorithm for linear MDP with low switching cost.
problem Large state space reinforcement learning problems with low switching cost.
method First algorithm for linear MDP with low switching cost, achieving near-optimal regret and switching cost.
result Regret bound of $\widetilde{O}\left(\sqrt{d^3H^4K}
ight)$ and near-optimal switching cost of $O\left(d H\log K
ight)$ .
New UCB algorithm for learning PSRs with tractable computation and accuracy.
problem Learning predictive state representations in sequential decision-making problems.
method Proposes a novel UCB-type algorithm with a bonus term to estimate PSRs accurately and efficiently.
result First known UCB-type approach for PSRs with guaranteed model accuracy and computational tractability.
Algorithm optimizes constrained reinforcement learning with dual variables.
problem Minimizing convex functional subject to convex constraint in large state spaces.
method VPDPO algorithm using Lagrangian and Fenchel duality.
result Achieves sublinear regret and constraint violation, globally optimal policy.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
ReLEX algorithm improves RL efficiency by selecting optimal representations.
problem Improving reinforcement learning efficiency through better representation selection.
method Proposes ReLEX algorithm for both online and offline RL, focusing on bilinear transition kernels.
result ReLEX algorithms achieve optimal or near-optimal performance in both online and offline RL settings.
Paper proposes an efficient RL algorithm for discounted MDPs using feature mapping.
problem Efficient reinforcement learning for large state and action spaces.
method Uses feature mapping to represent states and actions in a low-dimensional space, proposing a novel algorithm with polynomial regret bound.
result Achieves a O ( d T / ( 1 − γ ) 2 ) O(d\sqrt{T}/(1-γ)^2) O ( d T / ( 1 − γ ) 2 ) regret bound, near-optimal up to a ( 1 − γ ) − 0.5 (1-γ)^{-0.5} ( 1 − γ ) − 0.5 factor. Exploration in multi-task reinforcement learning is critical in training agents to deduce the underlying MDP. Many of the existing exploration frameworks such as E 3 E^3 E 3 , R m a x R_{max} R ma x , Thompson sampling assume a single stationary MDP and are not suitable for system identification in the multi-task setting. We present a nove…
We study the sample complexity of model-based reinforcement learning (henceforth RL) in general contextual decision processes that require strategic exploration to find a near-optimal policy. We design new algorithms for RL with a generic model class and analyze their statistical properties. Our algorithms have sample …
Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.
problem Challenging to learn accurate MDPs for high-dimensional states.
method Learn an abstract MDP over low-dimensional coarse states, using an abstraction function.
result Achieves superhuman performance on Pitfall! and higher reward with fewer samples.
Due to physiological variation, patients diagnosed with the same condition may exhibit divergent, but related, responses to the same treatments. Hidden Parameter Markov Decision Processes (HiP-MDPs) tackle this transfer-learning problem by embedding these tasks into a low-dimensional space. However, the original formul…
Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…
We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using low-dimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussi…
New algorithm REFUEL shows multitask representation learning is more sample-efficient in RL.
problem Understanding the benefit of representation learning in reinforcement learning.
method Developed REFUEL algorithm for multitask low-rank RL, analyzing both upstream and downstream tasks.
result Multitask representation learning is provably more sample-efficient than individual task learning.
We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is obs…
Improved exploration in adversarial MDPs via dilated bonuses.
problem Global exploration in adversarial MDPs with local-search policies.
method Dilated bonuses to facilitate global exploration.
result Improved regret bounds in various MDP settings.
Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.
problem Online learning in finite-horizon episodic MDPs with aggregate bandit feedback.
method Best-of-both-worlds (BOBW) algorithms using FTRL over occupancy measures, self-bounding techniques, and new loss estimators.
result First BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback achieving O ( log T ) O(\log T) O ( log T ) regret in stochastic and O ( T ) {O}(\sqrt{T}) O ( T ) regret in adversarial settings. We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…
Reinforcement learning (RL) in Markov decision processes (MDPs) with large state spaces is a challenging problem. The performance of standard RL algorithms degrades drastically with the dimensionality of state space. However, in practice, these large MDPs typically incorporate a latent or hidden low-dimensional structu…