Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.
problem Non-rectangular robust Markov decision processes under average-reward criterion.
method Proves history-dependent policies are robust-optimal, introduces transient-value framework, constructs epoch-based policy.
result Existence and properties of robust optimal policies, transient value bounds.
New method improves off-policy policy evaluation by balancing policy values.
problem Accurate off-policy policy evaluation in reinforcement learning.
method Developed an MDP model with balanced representation to estimate both individual and average policy values.
result Substantially lower Mean Squared Error (MSE) in various benchmarks and a real-world HIV treatment simulation.
New offline RL method handles average-reward MDPs with single-policy coverage.
problem Challenges in offline reinforcement learning due to distribution shift and non-uniform coverage.
method Develops an algorithm based on pessimistic discounted value iteration with quantile clipping.
result First fully single-policy sample complexity bound for average-reward offline RL.
This work tackles risk-sensitive deep RL by optimizing policies with variance constraints.
problem Risk and aleatoric uncertainty in deep reinforcement learning.
method Lagrangian and Fenchel dualities to transform the problem into an unconstrained saddle-point policy optimization problem, and an actor-critic algorithm to iteratively update policy, Lagrange multiplier, and Fenchel dual variable.
result The proposed actor-critic algorithm finds a globally optimal policy at a sublinear rate.
We consider the problem of sequential sampling from a finite number of independent statistical populations to maximize the expected infinite horizon average outcome per period, under a constraint that the expected average sampling cost does not exceed an upper bound. The outcome distributions are not known. We construc…
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
New RL algorithm tackles non-stationary environments with flexible policy updates.
problem Non-stationary reinforcement learning with time-varying rewards and transition probabilities.
method Model-free policy-based algorithm NS-NAC with restart-based exploration and dynamic learning rates.
result Dynamic regret of i l d e O ( ∣ S ∣ 1 / 2 ∣ A ∣ 1 / 2 Δ T 1 / 6 T 5 / 6 ) ilde{\mathscr O}(|S|^{1/2}|A|^{1/2}Δ_T^{1/6}T^{5/6}) i l d e O ( ∣ S ∣ 1/2 ∣ A ∣ 1/2 Δ T 1/6 T 5/6 ) for both algorithms. New algorithm achieves optimal regret in average reward MDPs without prior bias information.
problem Achieving optimal regret in average reward MDPs with computational efficiency and without prior bias information.
method Projective Mitigated Extended Value Iteration (PMEVI) to compute bias-constrained optimal policies efficiently.
result First tractable algorithm with minimax optimal regret of O ~ ( s p ( h ∗ ) S A T ) \widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T}) O ( sp ( h ∗ ) S A T ) . Enhances POLITEX for exploration in reinforcement learning with no-reward learning.
problem Learning to explore in reinforcement learning problems with no-reward.
method Modifies POLITEX to incorporate a pre-existing exploration policy.
result Achieves sublinear regret guarantees similar to POLITEX but without requiring all policies to explore.
Proposes a CVaR-based method to learn robust options in uncertain environments.
problem Learning options from inaccurate models with uncertain parameters.
method Extends CVaR-based policy gradient method to robust Markov decision processes.
result Produces options that perform well in both average and worst cases.
In the hypothesis of rare loss events, the general expression of the policy value has been determined as a functional of the "expected frequency / loss severity" function and of the retention function. Exponential disutility has been chosen after mathematical characterization of some of its economical aspects, where fu…
Procedure for unbiasedly estimating value of optimized policies.
problem Unbiased estimation of value of optimized policies in A/B testing.
method Bagging process with inverse-propensity-weighting and per-sample value estimates.
result Unbiased estimator of the value of deploying an optimized policy.
Backward exploration reduces sample complexity in policy evaluation.
problem Empirical policy evaluation in reinforcement learning.
method Backward exploration algorithms from high-cost states.
result Reduced average-case sample complexity to O ( log S ) O(\log S) O ( log S ) . The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
Kernel smoothing improves LLM reasoning efficiency.
problem Efficiently estimate value functions with limited samples for reinforcement learning.
method Kernelized advantage estimation using classical nonparametric statistics.
result Improved policy optimization with accurate value and gradient estimation.
In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation erro…
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
New RL algorithms improve average-reward performance.
problem Improving RL algorithms for average-reward criteria.
method Developed novel algorithms addressing average-reward criterion directly.
result ATRPO significantly outperforms TRPO in challenging MuJuCo environments.
Optimizes trading strategy considering alpha decay and transaction costs.
problem Maximizing reward in a multi-period portfolio with transaction costs and alpha decay.
method Formulated as an infinite horizon Markov Decision Process, solved using a modified value iteration algorithm with convergence proof and asymptotic analysis.
result Characterized optimal trading policy that maximizes average expected reward.
Variable renewables can avoid market value decline with policy changes.
problem Market value decline due to correlated generation from wind and solar.
method Theoretical analysis and simulation examples of market incentives and prices.
result Market value decline is due to policy assumptions, not inherent technology limitations.
We propose a fully distributed actor-critic algorithm approximated by deep neural networks, named \textit{Diff-DAC}, with application to single-task and to average multitask reinforcement learning (MRL). Each agent has access to data from its local task only, but it aims to learn a policy that performs well on average …
This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ ( Π ) n − 1 / 2 κ(Π)n^{-1/2} κ ( Π ) n − 1/2 , demonstrating substantial improvement over existing benchmarks. The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
End-to-end policy learning method improves CATE estimation.
problem Learning optimal treatment policies from partially observed data.
method Modified causal forest for policy learning.
result Maximizing policy value is equivalent to minimizing CATE.
Faster algorithms for solving multichain MDPs under average-reward criterion.
problem Navigating towards the best connected component in multichain MDPs.
method Developed algorithms to better solve the navigational subproblem, achieving faster convergence rates.
result Improved rates of convergence and sharper complexity measures for multichain MDPs.
Perry uses auxiliary data to estimate RL policy values with confidence intervals.
problem Leveraging auxiliary datasets for off-policy evaluation with uncertainty quantification.
method Two methods for constructing valid confidence intervals for OPE with data augmentation.
result Methods consistently produce confidence intervals covering ground truth policy values.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
RRPI improves offline RL by optimizing policies against worst-case dynamics.
problem Offline RL's performance degrades under distribution shift and transition uncertainty.
method Formulates offline RL as robust policy optimization, treating transition kernel as decision variable.
result RRPI achieves strong average performance on D4RL benchmarks, outperforming recent baselines.
New methods for off-policy evaluation in reinforcement learning.
problem Improving off-policy evaluation in reinforcement learning.
method Developed MWL and MQL estimators for importance weights and value functions.
result MWL and MQL provide a unified view of algorithms in reinforcement learning.
A new algorithm speeds up multi-agent reinforcement learning.
problem Complex interactions between agents in multi-agent reinforcement learning.
method Double averaging scheme for decentralized convex-concave saddle-point problems.
result The algorithm converges to the optimal solution at a global geometric rate.
Risk-sensitive RL aims to optimize performance while considering risk constraints or measures.
problem Optimizing expected value alone may not be sufficient; incorporating risk measures is necessary.
method Policy gradient search is used to solve risk-sensitive RL problems, considering various risk measures.
result Policy gradient methods can be adapted for risk-sensitive RL, addressing challenges and future directions.
Proposes three decentralized multi-agent reinforcement learning algorithms to reduce network congestion.
problem Finding a joint policy maximizing long-term return in a decentralized multi-agent system.
method Three fully decentralized multi-agent natural actor-critic (MAN) algorithms using linear function approximations.
result The proposed algorithms achieve performance comparable to or better than standard methods in reducing network congestion.
New RL method improves on standard discounted RL for operations research.
problem Applying RL to operations research problems, especially with non-zero rewards.
method Near-Blackwell-optimal RL algorithm that assesses average reward per step.
result Proves viability on challenging queuing system problems.
PGS uses neural networks to improve policies online without search trees.
problem Limited scalability of Monte Carlo Tree Search (MCTS) for high branching factor games.
method Adapts a neural network simulation policy via policy gradient updates, avoiding search trees.
result PGS achieves comparable performance to MCTS and defeats strong Hex agents.
AIF reformulated as convex MDP for adaptive behavior.
problem Adaptive behavior and policy optimization.
method Formulating AIF as convex MDP, deriving mirror descent algorithm.
result EFE minimization in AIF is equivalent to reward maximization in latent MDP, with epistemic component.
ScoreMatchingRiesz improves debiased machine learning and policy effects estimation.
problem Improving debiased machine learning and policy effects estimation.
method Score matching and Riesz representer estimation.
result Estimates policy path for continuous treatments, improving interpretability.
Enhanced tracking control for AUVs with improved policy gradient method.
problem Trajectory tracking problem for underactuated AUVs with unknown dynamics and constrained inputs.
method Hybrid actors-critics architecture with multiple actors and critics, Pseudo Q-learning, and deterministic policy gradient.
result High-level tracking control accuracy and stable learning of AUVs.
CADR estimator improves inference for contextual bandit data.
problem Valid inference on contextual bandit data.
method CADR estimator for policy value, addressing adaptive data collection challenges.
result CADR provides correct coverage of confidence intervals.
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD( λ λ λ ). result Improved numerical stability and broader step-size range.
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
Limited liability creates a conflict of interests between policyholders and shareholders of insurance companies. It provides shareholders with incentives to increase the risk of the insurer's assets and liabilities which, in turn, might reduce the value policyholders attach to and premiums they are willing to pay for i…
This work tackles learning Markov games with adversarial opponents and achieves both average reward and exploitation.
problem Achieving both average reward and exploiting adaptive opponents in Markov games.
method Develops efficient algorithms and proves hardness results for learning Markov games with adversarial opponents.
result Achieves K \sqrt{K} K -regret bounds for certain conditions on opponent policies, complemented by an exponential lower bound. A framework for reinforcement learning tackles CVRP with competitive results.
problem Optimizing routes for vehicles with limited capacity.
method Formulates action selection as a mixed-integer optimization problem, uses policy iteration to improve policies.
result Achieves an average gap of 1.7% with state-of-the-art OR methods on CVRP instances.
Unified q-learning for mean-field jump-diffusion models with unobservable population distribution.
problem Continuous-time q-learning in mean-field jump-diffusion models with unobservable population distribution.
method Proposed decoupled Iq-function for unified policy evaluation in MFG and MFC problems; unified q-learning algorithm based on test policies and averaged martingale orthogonality condition.
result Unified policy evaluation rule for MFG and MFC problems based on decoupled Iq-function.
STORM-PG uses momentum for faster policy gradient updates.
problem Improving policy gradient methods for reinforcement learning.
method Introduces STORM-PG, a SARAH-based algorithm with exponential moving average.
result Achieves O ( 1 / ε 3 ) O(1/ε^3) O ( 1/ ε 3 ) sample complexity, matching best-known rate. Method bounds continuous-valued treatment effects when confounding variables are hidden.
problem Inferring causal effects of continuous treatments when hidden confounders are present.
method Novel methodology to bound average and conditional average continuous-valued treatment effects.
result Method gives tighter coverage of true dose-response curve than existing methods.
This paper solves steady-state planning for multichain MDPs.
problem Specifying constraints on the steady-state behavior of an agent.
method Linear programming solution for multichain MDPs.
result Optimal solutions yield stationary policies with rigorous guarantees.
AAPI improves regret bound for undiscounted continuing learning in uniformly ergodic MDPs.
problem Improving regret bounds for undiscounted continuing learning in uniformly ergodic MDPs.
method Adaptive approximate policy iteration (AAPI) with online learning techniques and data-dependent adaptive learning rate.
result AAPI achieves a i l d e O ( T 2 / 3 ) ilde{O}(T^{2/3}) i l d e O ( T 2/3 ) regret bound, improving over the best existing bound of i l d e O ( T 3 / 4 ) ilde{O}(T^{3/4}) i l d e O ( T 3/4 ) .