New algorithms reduce dynamic regret in online MDPs with changing losses.
problem Online MDPs with adversarial loss changes and known transitions.
method Dynamic regret measure, novel ensemble algorithms for three models.
result Provably optimal dynamic regret bounds for episodic SSP, improved bounds for predictable environments.
New RL algorithm tackles online robust MDPs with uncertainty.
problem Developing robust reinforcement learning models for real-world environments.
method Proposes a robust optimistic policy optimization algorithm for online robust MDPs.
result Establishes the first regret bound for online robust MDPs.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Algorithm learns from offline data to improve performance in target environment.
problem Learning from offline data in a target environment with unknown shifts.
method Adaptive algorithm that uses offline data to improve performance when informative.
result Algorithm provably improves performance over purely online learning when offline data are informative.
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
New RL approach uses resets to learn complex MDPs efficiently.
problem Learning complex MDPs with high-dimensional states and function approximation.
method Local simulator access and recursive value function search.
result Proven sample-efficient learning for MDPs with low coverability.
Algorithm for online learning in MDPs with linear function approximation and bandit feedback.
problem Online learning in MDPs with changing reward functions and limited feedback.
method Developed MDP-LinExp3 algorithm with theoretical guarantees.
result Proved regret bounds for MDP-LinExp3 algorithm.
New algorithm for online learning in episodic MDPs with convex objectives.
problem Online episodic convex reinforcement learning.
method Online mirror descent algorithm with varying constraint sets and exploration bonus.
result Near-optimal regret bounds for online CURL without prior knowledge of transition function.
New coverage conditions improve sample efficiency in online reinforcement learning.
problem Improving sample efficiency in online reinforcement learning with function approximation.
method Identifying and studying new coverage conditions for online reinforcement learning.
result Improved regret bounds achieved with new coverage conditions.
Paper proposes an efficient online learning method using an offline dataset for infinite horizon MDPs.
problem Efficient online reinforcement learning in infinite horizon MDPs with an unknown expert policy.
method Bayesian approach to model the expert's policy and minimize cumulative regret.
result Upper bound on regret of i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) for the Informed PSRL algorithm. New complexity measure helps in agnostic reinforcement learning with or without access to MDP dynamics.
problem Understanding the number of rounds needed to learn an ε-suboptimal policy in unknown MDPs.
method Introducing spanning capacity as a new complexity measure and developing POPLER algorithm.
result There is a separation between generative and online access models for agnostic learnability.
This work tackles representation learning for RL in low-rank MDPs, improving sample efficiency.
problem How to learn a compact representation for RL in low-rank MDPs efficiently.
method Proposes REP-UCB for online RL and develops an algorithm for offline RL under partial coverage.
result Significantly improved sample complexity for online RL and competitive performance for offline RL.
New algorithms reduce regret in online MDPs by adapting to data and variance.
problem Adapting to both adversarial and stochastic environments in online MDPs.
method Develops algorithms based on global optimization and policy optimization, using optimistic follow-the-regularized-leader with log-barrier regularization.
result Achieves refined data-dependent and variance-dependent regret bounds.
Study non-asymptotic BPI guarantees for online RL.
problem Identify optimal policy in MDP with high confidence.
method Non-asymptotic sample complexity guarantees for NaS algorithm.
result Sample complexity depends on MDP connectivity and curvature.
We consider the recently proposed reinforcement learning (RL) framework of Contextual Markov Decision Processes (CMDP), where the agent interacts with a (potentially adversarial) sequence of episodic tabular MDPs. In addition, a context vector determining the MDP parameters is available to the agent at the start of eac…
We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show O ~ ( L ∣ X ∣ ∣ A ∣ T ) \tilde{O}(L|X|\sqrt{|A|T}) O ~ ( L ∣ X ∣ ∣ A ∣ T ) regret bound, where T T T is the number of episodes, X X X is the state space, $A…
New L 1 L_1 L 1 -Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces L 1 L_1 L 1 -Coverage objective to enable efficient exploration and planning. result First computationally efficient algorithms for online reinforcement learning with low coverability.
Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.
problem Online learning in finite-horizon episodic MDPs with aggregate bandit feedback.
method Best-of-both-worlds (BOBW) algorithms using FTRL over occupancy measures, self-bounding techniques, and new loss estimators.
result First BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback achieving O ( log T ) O(\log T) O ( log T ) regret in stochastic and O ( T ) {O}(\sqrt{T}) O ( T ) regret in adversarial settings. Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
Improved online Q-learning for MDPs with concentration bounds.
problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.
method Smoothed ε n ε_n ε n -Greedy exploration scheme combining ε n ε_n ε n -greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation. result Near- i l d e O ( N 9 / 10 ) ilde{O}(N^{9/10}) i l d e O ( N 9/10 ) regret bound for Smoothed ε n ε_n ε n -Greedy exploration scheme. Paper analyzes online reinforcement learning with outcome-based feedback, providing efficient algorithms and fundamental limits.
problem Assigning credit to actions in reinforcement learning with only endpoint rewards.
method Develops a provably sample-efficient algorithm for online reinforcement learning with general function approximation.
result Achieves O ( C m c o v H 3 / ε 2 ) O(C_{
m cov} H^3/ε^2) O ( C m co v H 3 / ε 2 ) sample complexity, characterizing statistical separation between outcome-based and per-step rewards. Efficiently identifies best policies in tabular MDPs with reduced computational cost.
problem Identifying the best policy in tabular MDPs with high computational cost.
method Combines posterior sampling with online learning to achieve asymptotic optimality.
result Achieves optimal sample complexity and posterior contraction rate with O ( S 2 A H ) O(S^2AH) O ( S 2 A H ) per episode. AAPI improves regret bound for undiscounted continuing learning in uniformly ergodic MDPs.
problem Improving regret bounds for undiscounted continuing learning in uniformly ergodic MDPs.
method Adaptive approximate policy iteration (AAPI) with online learning techniques and data-dependent adaptive learning rate.
result AAPI achieves a i l d e O ( T 2 / 3 ) ilde{O}(T^{2/3}) i l d e O ( T 2/3 ) regret bound, improving over the best existing bound of i l d e O ( T 3 / 4 ) ilde{O}(T^{3/4}) i l d e O ( T 3/4 ) . We study online learning of finite Markov decision process (MDP) problems when a side information vector is available. The problem is motivated by applications such as clinical trials, recommendation systems, etc. Such applications have an episodic structure, where each episode corresponds to a patient/customer. Our ob…
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
Online Apprenticeship Learning aims to match expert performance without access to cost functions.
problem Learning an agent's policy to match expert performance in an MDP without cost function access.
method Combines mirror descent based no-regret algorithms for policy optimization and cost learning, with optimistic exploration.
result Derives an algorithm with O ( K ) O(\sqrt{K}) O ( K ) regret, practical for high-dimensional control problems. New RL approach learns dynamic VCG mechanisms in unknown MDP environments.
problem Learning dynamic VCG mechanisms in unknown MDP environments.
method Reward-free online RL for exploration, combined with function approximation.
result Regret bound of O ~ ( T 2 / 3 ) \tilde{\mathcal{O}}(T^{2/3}) O ~ ( T 2/3 ) for dynamic VCG mechanism learning. New insights show coverage conditions are crucial for efficient online reinforcement learning.
problem The role of coverage conditions in determining sample complexity of offline reinforcement learning.
method Established a connection between coverage conditions and sample efficiency in online reinforcement learning.
result Coverability, a structural property of MDPs, enables sample-efficient exploration in online reinforcement learning.
Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is b…
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
The paper introduces a method to learn and apply value envelopes for faster online reinforcement learning.
problem Accelerating online reinforcement learning using offline data with theoretical grounding.
method A two-stage framework: offline data for learning value bounds, online algorithms for applying them.
result Substantial regret reductions in empirical tests on tabular MDPs.
Adversarial online multi-task RL with task separation.
problem Minimize regret in an adversarial online multi-task setting with unknown MDPs.
method Prove minimax and instance-specific lower bounds, develop a clustering algorithm with optimal sample complexity and regret.
result Tight sample complexity and regret bounds for adversarial online multi-task RL.
Significant improvements in regret analysis for adaptive online learning problems.
problem Exploiting low variance in online learning problems without known variances.
method Novel peeling-based regret analysis leveraging elliptical potential `count` lemma.
result Significant improvements in regret bounds for linear bandits and linear mixture MDPs.
A new model-free algorithm achieves near-optimal regret for infinite-horizon MDPs.
problem Model-free reinforcement learning for infinite-horizon average-reward MDPs.
method Exploration Enhanced Q-learning (EE-QL) for weakly communicating MDPs.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret bound for general weakly communicating MDPs. New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
Fine-tuning RL with offline data reduces online interactions.
problem Optimizing RL with limited online interactions and offline data.
method Developed algorithm extsc{FTPedel} for MDPs with linear structure.
result Optimally reduces the number of online interactions needed.
New algorithm reduces sample complexity for safe reinforcement learning.
problem Safe reinforcement learning in constrained MDPs with performance and safety constraints.
method Model-based primal-dual algorithm balancing regret and bounded constraint violations.
result Proves near-optimal policies with bounded violations or zero violations in CMDPs.
Online learning algorithms are designed to perform in non-stationary environments, but generally there is no notion of a dynamic state to model constraints on current and future actions as a function of past actions. State-based models are common in stochastic control settings, but commonly used frameworks such as Mark…
Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision making under uncertainty. The classical approaches for solving MDPs are well known and have been widely studied, some of which rely on approximation techniques to solve MDPs with large state space and/or action space. However…
Improved algorithm detects changes in RL environments with non-stationary MDPs.
problem Learning in non-stationary reinforcement learning environments.
method R-BOCPD-UCRL2 algorithm for MDPs with multinomial state transitions.
result Near-optimal theoretical guarantees in terms of false-alarm rate and detection delay.
This work explores efficient reinforcement learning with density features in low-rank MDPs.
problem Efficient reinforcement learning with density features in low-rank MDPs.
method Proposes algorithms for off-policy estimation and online construction of exploratory data distributions.
result Demonstrates sample-efficient learning with density features in low-rank MDPs, overcoming technical challenges.
Develops RL algorithm for non-Markovian, non-stationary reward streams.
problem Maximizing rewards from non-Markovian, non-stationary reward streams.
method Uses causal DAG to construct Markov states, solves periodic MDP.
result Optimal state construction maximizes discounted rewards.
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
Efficiently selects seed nodes to maximize content influence in unknown social networks.
problem Maximizing content spread in social networks with unknown network model.
method Formulated as an infinite-horizon discounted MDP, uses model-based reinforcement learning to select seed users adaptively.
result Established a regret bound of O ~ ( T ) \widetilde O(\sqrt{T}) O ( T ) for the algorithm. Paper establishes first instance-dependent lower bound for PAC reinforcement learning.
problem Identifying near-optimal policies in tabular MDPs with minimal samples.
method Proposes instance-dependent lower bound for sample complexity.
result Lower bound closely matches PEDEL algorithm's sample complexity.
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is obs…
TempLe learns transition templates for efficient multi-task RL.
problem Efficiently transferring knowledge across different RL tasks with varying state/action spaces.
method Generates transition dynamics templates to abstract similarities between tasks.
result Achieves significantly lower sample complexity than single-task or multi-task methods.