Paper eliminates warm-up phase for PO in linear MDPs, achieving optimal regret.
problem Costly warm-up phase in PO algorithms for linear MDPs.
method Simple contraction mechanism replaces warm-up phase.
result Achieves rate-optimal regret with improved dependence on problem parameters.
Extends linear MDP to handle nonlinear rewards.
problem Restrictive linear MDP assumption limits real-world applicability.
method Proposes Generalized Linear MDP (GLMDP) with GLMs for rewards.
result Develops offline RL algorithms achieving suboptimality guarantees.
Algorithm learns mixtures of Markov chains and MDPs from short trajectories.
problem Learning mixtures of Markov chains and MDPs from short unlabeled trajectories.
method Subspace estimation, spectral clustering, EM algorithm, model estimation, classification.
result 96.6% average accuracy on a mixture of two MDPs in gridworld, outperforming EM algorithm with random initialization.
A new algorithm finds optimal solutions for constrained decision processes.
problem Optimizing state-value functions with constraints in CMDPs.
method Gradient-Aware Search (GAS) exploiting PWLC structure.
result GAS converges faster and more reliably than existing methods.
A new MDP with Bandits approach for sequential decision making in linear-flow scenarios.
problem Sequential decision making with limited feedback in a linear-flow context.
method Formulated as an MDP with Bandits, using Thompson sampling for action selection and exact dynamic programming for allocation.
result The proposed MDP with Bandits algorithm outperforms other methods in sequential decision making.
A new framework for robust policy learning in MDPs with linear mixture dynamics.
problem Off-dynamics challenge in real-world decision-making problems.
method Linear mixture DRMDP framework, meta algorithm for robust policy learning.
result The new framework provides a more refined representation of uncertainties.
Stochastic domains often involve risk-averse decision makers. While recent work has focused on how to model risk in Markov decision processes using risk measures, it has not addressed the problem of solving large risk-averse formulations. In this paper, we propose and analyze a new method for solving large risk-averse …
Paper improves TD learning algorithm bounds with linear approx.
problem Sharp bounds for TD method performance in MDPs.
method Polyak-Ruppert averaging, universal step size, refined error bounds, stability of random matrices.
result Near-optimal variance and bias terms achieved.
Paper presents an efficient algorithm for linear MDP with low switching cost.
problem Large state space reinforcement learning problems with low switching cost.
method First algorithm for linear MDP with low switching cost, achieving near-optimal regret and switching cost.
result Regret bound of $\widetilde{O}\left(\sqrt{d^3H^4K}
ight)$ and near-optimal switching cost of $O\left(d H\log K
ight)$ .
A new method for risk-averse decision-making in Markov processes with improved regret bounds.
problem Risk-averse decision-making in Markov processes.
method Introduces mini-batch measures and multipattern risk-averse problems in a feature-based Q Q Q -learning method. result Proves a high-probability regret bound of O ( H 2 N H K ) \mathcal{O}\big(H^2 N^H \sqrt{ K}\big) O ( H 2 N H K ) for the Q Q Q -learning method. Introduces DF framework for sampling decisions from target distributions.
problem Sampling from target distributions with additional guidance.
method DF framework based on MDP and Path Integral Diffusion.
result DF enhances guided sampling across various applications.
New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.
problem Solving Markov decision processes with uncertain transition probabilities.
method Distributionally robust Q Q Q -learning algorithm for Wasserstein uncertainty. result Convergence of the algorithm proved and demonstrated with real data.
Algorithm for online decision making with unknown dynamics and aggregate feedback.
problem Online decision making with unknown dynamics and aggregate bandit feedback.
method Developed an algorithm based on online mirror descent with a self-concordant barrier regularization and an increasing learning rate schedule.
result Achieved O ( K ) O(\sqrt{K}) O ( K ) regret for the online Markov Decision Process with K K K episodes. We consider the problem of computing optimal policies in average-reward Markov decision processes. This classical problem can be formulated as a linear program directly amenable to saddle-point optimization methods, albeit with a number of variables that is linear in the number of states. To address this issue, recent …
Develops new reinforcement learning methods for complex constrained decision-making problems.
problem Complex constrained decision-making problems with a continuum of constraints.
method Proposes semi-infinitely constrained Markov decision processes (SICMDPs) and two reinforcement learning algorithms: SI-CRL and SI-CPO.
result Demonstrates the effectiveness of SI-CRL and SI-CPO in solving complex sequential decision-making tasks.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where ba…
DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.
problem Policy optimization in RL under distribution shift and adversarial dynamics.
method DR-RPO algorithm incorporating reference-policy regularization and upper confidence bonus for exploration.
result DR-RPO achieves sublinear regret and polynomial suboptimality bounds in robust RL.
This paper proposes an active sampling method for meta-learning using MDPs.
problem Exploiting relationships between tasks and classes in meta-learning.
method Formulates the problem as a MDP, uses UCB, Gittins Index, and linear programming solutions.
result Significant reductions in sample complexity for active selection schemes.
New algorithms for learning MDPs with linear approximations in infinite-horizon settings.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method Optimism principle, adversarial linear bandits, Natural Policy Gradient.
result Efficient algorithms with optimal or near-optimal regret bounds.
New algorithm reduces switching costs in RL beyond linear MDPs.
problem Costly policy switching in reinforcement learning.
method ELEANOR-LowSwitching algorithm for linear Bellman-complete MDPs.
result Achieves near-optimal regret with logarithmic switching cost.
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
Paper tackles non-monotonic resource utilization in sequential decision-making.
problem Sequential decision-making under uncertainty with resource constraints.
method Introduces a new MDP policy with constant regret against LP relaxation.
result Develops a learning algorithm with logarithmic regret for unknown outcome distributions.
New method for optimistic planning in MDPs using regularization.
problem Optimistic planning in infinite-horizon discounted MDPs.
method Regularized dynamic programming for approximate value iteration.
result Achieves near-optimal statistical guarantees in learning policies.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret K ( M ) log ( T ) + o ( log ( T ) ) K(M) \log(T) + \mathrm{o}(\log(T)) K ( M ) log ( T ) + o ( log ( T )) for communicating Markov decision processes. AIF reformulated as convex MDP for adaptive behavior.
problem Adaptive behavior and policy optimization.
method Formulating AIF as convex MDP, deriving mirror descent algorithm.
result EFE minimization in AIF is equivalent to reward maximization in latent MDP, with epistemic component.
New RL algorithm for linear MDPs with nearly optimal regret.
problem Optimizing reinforcement learning for linear mixture Markov decision processes.
method Proposed a new Bernstein-type concentration inequality for self-normalized martingales and a computationally efficient algorithm UCRL-VTR+.
result UCRL-VTR+ achieves nearly minimax optimal regret of i l d e O ( d H T ) ilde O(dH\sqrt{T}) i l d e O ( d H T ) . This paper optimizes MDP policies for efficient state aggregation.
problem Optimizing policies in aggregated Markov chains while preserving optimal performance.
method Homomorphic mappings to establish optimal policy equivalence and derive performance bounds.
result Developed HPG and EBHPG methods for efficient aggregation and policy optimization.
Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
Study on distributional TD learning with linear approximations for better return estimation.
problem Estimating the return distribution of a policy in reinforcement learning.
method Finite-sample analysis of distributional TD learning with linear function approximation, using the linear-categorical Bellman equation and exponential stability arguments for products of random matrices.
result Sample complexity of linear distributional TD learning matches that of classic linear TD learning, indicating similar difficulty in estimating return distribution versus its expectation.
New RL algorithm achieves nearly optimal performance for linear MDPs.
problem Optimal reinforcement learning for episodic linear MDPs.
method Weighted linear regression with variance estimator and rare-switching policy.
result Achieves nearly minimax optimal regret i l d e O ( d H 3 K ) ilde O(d\sqrt{H^3K}) i l d e O ( d H 3 K ) . Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an…
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
New algorithms identify best policies in discounted linear MDPs efficiently.
problem Identifying the best policy in discounted linear MDPs with limited samples.
method Derive lower bounds and devise simple yet near-optimal algorithms.
result Upper bound on sample complexity matches existing bounds.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
Paper introduces a new method for improving reinforcement learning performance using transfer learning.
problem Improving reinforcement learning performance with limited sample sizes in dynamic decision-making scenarios.
method Developed a novel ``re-weighted targeting procedure'' and ``transfer deep Q ∗ Q^* Q ∗ -learning'' approach. result Demonstrated improved reinforcement learning performance through strategic sample construction.
Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a i l d e O ( d 3 / 4 H 2 K 3 / 4 ) ilde{\mathcal{O}}(d^{3/4}H^2K^{3/4}) i l d e O ( d 3/4 H 2 K 3/4 ) regret bound. Agents collaboratively learn optimal policies in MDPs with limited capabilities.
problem Learning optimal policies in MDPs with heterogeneous agents and limited communication.
method Introduced concepts of leakage probabilities and proposed Federated-Q protocol (FedQ) for collaborative learning.
result FedQ protocol effectively aggregates knowledge and modifies learning problems for further training.
Paper proposes a privacy-preserving RL algorithm for linear MDPs with theoretical guarantees.
problem Protecting users' private data in personalized services using RL.
method Local differential privacy (LDP) for RL with linear function approximation.
result Achieves a regret bound of $O(d^{5/4}H^{7/4}T^{3/4}\left(\log(1/δ)
ight)^{1/4}\sqrt{1/\varepsilon})$ for linear mixture MDPs.
Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.
problem Finding safe zones in policy Markov Decision Processes to limit trajectory escape.
method Bi-criteria approximation learning algorithm with polynomial sample complexity.
result Achieves almost 2 approximation for both escape probability and safe zone size.
Deep nets solve MDPs without high dimensions.
problem Solving Bellman equations for MDPs in high dimensions.
method Deep neural networks with ReLU activation approximating payoff and transition functions.
result Deep nets can approximate Q Q Q -functions in polynomially bounded parameters. New method uses reinforcement learning to sample from complex data structures efficiently.
problem Constructing reliable samples from high-dimensional polytopes for goodness-of-fit tests.
method Markov decision process and reinforcement learning for sampling.
result Demonstrated scalable tools from linear algebra for theoretical guarantees in non-linear algebra context.
Risk measures applied to dynamic Markov processes with varying risk aversion.
problem Investigating dynamic risk measures in Markov decision processes with varying risk aversion.
method Distributional viewpoint on law-invariant convex risk measures, applied to Markov decision processes with latent costs and random actions.
result Existence of optimal policies in finite and infinite time horizons under mild assumptions.