The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
The paper tackles reinforcement learning with exogenous variables and rewards.
problem Exogenous state variables and rewards slow reinforcement learning by introducing uncontrolled variation.
method Formalizes exogenous state variables and rewards, decomposes MDP into exogenous and endogenous components, and introduces algorithms to discover these components.
result Optimal policies for the endogenous MDP are also optimal for the original MDP, but the endogenous MDP is easier to solve due to reduced variance.
New complexity measure MEHC refines MDP upper bounds and rewards informativeness.
problem Refining complexity measures for MDPs and understanding reward informativeness.
method Introducing MEHC, a new complexity measure that tightens MDP diameter by accounting for reward structure.
result MEHC replaces diameter in upper bounds on optimal value span and UCRL2-like algorithms' regret.
A new estimator for state values in reinforcement learning reduces complexity and improves convergence.
problem Estimating state values in reinforcement learning with Markov reward processes.
method Loop estimator exploiting regenerative structure of Markov reward processes.
result Instance-dependent convergence rate of O ~ ( τ s / T ) \widetilde{O}\left(\sqrt{τ_s/T}\right) O ( τ s / T ) for estimating state values. Detects spiky corruption in CRMDPs to learn optimal policies.
problem Learning optimal policies in environments with imperfect reward functions.
method Characterized spiky reward corruption, introduced algorithm to detect corrupt states.
result Algorithm can detect corrupt states and learn optimal policies.
Extends linear MDP to handle nonlinear rewards.
problem Restrictive linear MDP assumption limits real-world applicability.
method Proposes Generalized Linear MDP (GLMDP) with GLMs for rewards.
result Develops offline RL algorithms achieving suboptimality guarantees.
New Q-learning method achieves optimal sample complexity for average-reward problems.
problem Challenges in achieving optimal sample complexity for average-reward Q-learning.
method Synchronous and asynchronous Q-learning with a new contraction principle.
result Optimal O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) sample complexity guarantees. The paper studies reward concentration in MDPs, covering asymptotic and non-asymptotic settings.
problem Reward concentration in Markov Decision Processes (MDPs).
method Unified approach to reward concentration in MDPs, including asymptotic and non-asymptotic bounds.
result Rate-equivalent definitions of regret for learning policies.
ARL uses queries to learn rewards, focusing on cost vs. reward value.
problem How to efficiently use queries to learn rewards in reinforcement learning.
method Proposed and evaluated heuristic approaches for ARL in multi-armed bandits and MDPs.
result Challenging aspects of ARL highlighted, including intractability of value computation.
Paper models MOOC users behavior using MDP for easier classification.
problem Difficulty in defining and classifying MOOC users based on their behavior.
method Uses Markov Decision Process (MDP) to model user behavior from log data.
result Classifies users based on their intentions inferred from MDP rewards.
Exogenous state variables and rewards can slow down reinforcement learning by injecting uncontrolled variation into the reward signal. We formalize exogenous state variables and rewards and identify conditions under which an MDP with exogenous state can be decomposed into an exogenous Markov Reward Process involving on…
New algorithm handles MDPs with unknown, changing rewards efficiently.
problem Handling MDPs with unknown, changing rewards in large state spaces.
method Developed an algorithm with O ( τ ( ln ∣ S ∣ + ln ∣ A ∣ ) T ln ( T ) ) O(\sqrt{τ(\ln|S|+\ln|A|)T}\ln(T)) O ( τ ( ln ∣ S ∣ + ln ∣ A ∣ ) T ln ( T )) regret bound and a modified algorithm with polynomial complexity. result Achieved state-of-the-art regret bounds for large scale MDPs with changing rewards.
Study of multi-armed bandits with state-switching rewards using Markov models.
problem Multi-armed bandit problem with state-switching rewards.
method Spectral method-of-moments estimations for hidden Markov models, belief error control, upper-confidence-bound methods.
result Upper bound of O ( T 2 / 3 log T ) O(T^{2/3}\sqrt{\log T}) O ( T 2/3 log T ) for the learning algorithm performance. Two new algorithms improve model-free RL for infinite-horizon MDPs.
problem Learning in infinite-horizon average-reward MDPs.
method Two model-free algorithms for infinite-horizon average-reward MDPs.
result Improved regret bounds for model-free RL in MDPs.
Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
We optimize saddle-point problems for large-scale Markov decision processes.
problem Optimizing policies in large-scale Markov decision processes.
method Characterized conditions for convergence and designed an optimization algorithm.
result Our algorithm converges faster and is state-space independent.
This work extends reinforcement learning to handle non-cumulative objectives.
problem Optimizing functions of rewards rather than their sum in decision processes.
method Mapping NCMDPs to standard MDPs for reinforcement learning.
result Reinforcement learning techniques can be applied to NCMDPs.
Reward suffices for convex MDPs, expanding RL to new problems.
problem Capturing goals as convex functions of stationary distribution.
method Reformulated as a min-max game using Fenchel duality.
result Convex MDPs require non-stationary reward functions.
Logarithmic regret for continuous-time reinforcement learning.
problem Continuous-time Markov decision processes with unknown transition probabilities and holding times.
method Upper confidence reinforcement learning, mean holding time estimation, stochastic comparison of point processes.
result Logarithmic regret bound achieved in finite time.
UCRL-CMDP algorithm optimizes RL with constraints on average costs.
problem Optimizing RL in MDPs with average cost constraints.
method Model-based RL algorithms maximizing reward while keeping costs within bounds.
result UCRL-CMDP algorithm's expected regret is upper-bounded by $T^{2\slash 3}$ .
New RL method explores environments without rewards, achieving efficient policy generation.
problem Efficiently exploring unknown environments without predefined rewards.
method Optimistic value-iteration algorithm with kernel and neural function approximations.
result Achieves O ~ ( 1 / ε 2 ) \widetilde{\mathcal{O}}(1 /\varepsilon^2) O ( 1/ ε 2 ) sample complexity for generating policies or equilibria. New algorithm learns optimal policy for average reward MDPs with sample complexity matching lower bound.
problem Learning optimal policy for average reward in uniformly ergodic MDPs.
method Developed an estimator with sample complexity of O(|S||A|t_{mix}ε^{-2}).
result First algorithm to match lower bound of existing literature.
We consider reinforcement learning in changing Markov Decision Processes where both the state-transition probabilities and the reward functions may vary over time. For this problem setting, we propose an algorithm using a sliding window approach and provide performance guarantees for the regret evaluated against the op…
AIF reformulated as convex MDP for adaptive behavior.
problem Adaptive behavior and policy optimization.
method Formulating AIF as convex MDP, deriving mirror descent algorithm.
result EFE minimization in AIF is equivalent to reward maximization in latent MDP, with epistemic component.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
The paper introduces a method to learn Markov state abstractions for reinforcement learning.
problem Learning Markov state representations in complex environments.
method The paper introduces a novel set of conditions and a training procedure combining inverse model estimation and temporal contrastive learning.
result The approach learns representations that capture the underlying structure of the domain and improve sample efficiency.
We characterize value functions in partially observable MDPs as semi-algebraic sets.
problem Understanding feasible value functions in partially observable Markov decision processes.
method Characterization of feasible value functions as semi-algebraic sets defined by polynomial inequalities.
result The feasible set of value functions in POMDPs is a semi-algebraic set, not a polytope as in MDPs.
New algorithm achieves optimal regret in average reward MDPs without prior bias information.
problem Achieving optimal regret in average reward MDPs with computational efficiency and without prior bias information.
method Projective Mitigated Extended Value Iteration (PMEVI) to compute bias-constrained optimal policies efficiently.
result First tractable algorithm with minimax optimal regret of O ~ ( s p ( h ∗ ) S A T ) \widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T}) O ( sp ( h ∗ ) S A T ) . Paper estimates risks in MDPs using state lumping and SAT, showing its effectiveness.
problem Estimating risks in Markov decision processes with state augmentation.
method State augmentation transformation, isotopic states, and state lumping.
result SAT and state lumping effectively estimate mean-variance and exponential utility risks.
Modeling decision-making dynamics in MDD patients using RL-HMM.
problem Characterize reward learning strategies in MDD patients.
method Proposed RL-HMM framework to analyze reward-based decision-making.
result MDD patients show reduced engagement in RL compared to healthy controls.
Efficient algorithm for learning from indirect feedback in complex decision-making scenarios.
problem Learning from indirect feedback in realistic scenarios with personalized mechanisms.
method IGW algorithm for policy optimization, extending reward-estimator construction from single-step to multi-step.
result Achieves sublinear regret guarantee for contextual episodic MDPs with personalized feedback.
New algorithm optimizes multi-objective outcomes in uncertain environments.
problem Optimizing global concave rewards in online Markov decision processes with multiple actions.
method No-regret algorithm based on online convex optimization and UCRL2, with a gradient threshold procedure.
result Non-stationary policy diversifies outcomes to optimize the global concave reward.
Paper investigates IRL for learning expert agents' reward functions in LOB dynamics.
problem Learning expert agents' reward functions in LOB dynamics.
method Investigates IRL methods to infer reward functions from expert demonstrations in LOB environments.
result GP-based and BNN methods can discover non-linear reward functions in LOB dynamics.
New algorithm for reward-free RL with linear function approximation, reducing sample complexity.
problem Efficiently learning optimal policies without prior reward information in complex environments.
method Developed an algorithm for reward-free RL in linear Markov decision processes, proving sample complexity bounds.
result Polynomial sample complexity in feature dimension and planning horizon, independent of states and actions.
Faster algorithms for solving multichain MDPs under average-reward criterion.
problem Navigating towards the best connected component in multichain MDPs.
method Developed algorithms to better solve the navigational subproblem, achieving faster convergence rates.
result Improved rates of convergence and sharper complexity measures for multichain MDPs.
Algorithm optimizes decision-making in unknown MDPs with minimal regret.
problem Optimizing decision-making in unknown discrete MDPs with bounded expected shortest path.
method Developed BUCRL{} algorithm achieving i l d e O ( D S A T ) ilde{\mathcal{O}}(\sqrt{DSAT}) i l d e O ( D S A T ) regret. result First polynomial time Bayesian algorithm for unknown MDPs with high probability worst-case regret.
Overfitting occurs when RL agents correlate rewards with spurious observation features.
problem Overfitting in reinforcement learning due to correlation with spurious observation features.
method Developed a framework to analyze and design synthetic benchmarks from modified observation spaces.
result Agents can overfit to different observation spaces even if the MDP dynamics are fixed.
This paper presents four different ways of looking at the well-known Least Squares Temporal Differences (LSTD) algorithm for computing the value function of a Markov Reward Process, each of them leading to different insights: the operator-theory approach via the Galerkin method, the statistical approach via instrumenta…
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret K ( M ) log ( T ) + o ( log ( T ) ) K(M) \log(T) + \mathrm{o}(\log(T)) K ( M ) log ( T ) + o ( log ( T )) for communicating Markov decision processes. New RL method tackles dynamic MDPs with evolving rewards and states.
problem Dynamic MDPs with evolving rewards and states.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).
result Achieves dynamic regret bound for non-stationary MDPs.
Proposes CWAE-IRL for efficient IRL in complex environments.
problem Learning reward functions from expert actions in MDPs.
method Variational inference with CVAE and Wasserstein loss.
result CWAE-IRL learns reward functions efficiently in complex environments.
Study ℓ ∞ \ell_\infty ℓ ∞ bounds for MRP value function estimation.
problem Estimate MRP value function from samples.
method Analyze standard and robust plug-in approaches, establish bounds.
result Non-asymptotic and data-dependent ℓ ∞ \ell_\infty ℓ ∞ -norm bounds. This work improves Q-learning for average-reward MDPs, reducing sample and communication complexities in federated settings.
problem Improving sample complexity of Q-learning for average-reward MDPs.
method Simple Q-learning algorithm with carefully chosen parameters for both single-agent and federated scenarios.
result Established first federated Q-learning algorithm for average-reward MDPs with provable efficiency in sample and communication complexities.
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
New algorithm learns safe policies in unknown environments.
problem Learning safe policies in unknown, potentially unsafe environments.
method C-UCRL: Upper Confidence Reinforcement Learning for constrained MDPs.
result Achieves sub-linear regret while satisfying constraints.
In reinforcement learning (RL) , one of the key components is policy evaluation, which aims to estimate the value function (i.e., expected long-term accumulated reward) of a policy. With a good policy evaluation method, the RL algorithms will estimate the value function more accurately and find a better policy. When th…
TLRS improves predictive power of mined formulaic alpha factors.
problem Sparse rewards in RL for mining formulaic alpha factors.
method Trajectory-level Reward Shaping (TLRS) with reward centering.
result TLRS boosts predictive power by 9.29% over existing methods.