Improves RL generalization by minimizing adversarial risk.
problem Overfitting to training environments and poor generalization to unseen scenarios.
method Introduces minimax formulation and distributional framework to RL.
result Trained policy shows improved generalization to different environments.
Study minimax off-policy evaluation in multi-armed bandits with known and unknown behavior policies.
problem Evaluate policies in multi-armed bandits with unknown behavior policies.
method Develop minimax rate-optimal procedures for known and unknown behavior policies, including the Switch estimator and Chebyshev polynomial-based estimator.
result Plug-in estimator achieves optimal competitive ratio up to a logarithmic factor when behavior policy is unknown.
Unified minimax value interval for off-policy evaluation and optimization.
problem Overcoming the exponential variance in off-policy evaluation and policy optimization.
method Unified minimax value interval using marginalized importance weights.
result Unified value interval with double robustness, valid when either value-function or importance-weight class is well specified.
The paper tackles robust policy learning from multiple data sources.
problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.
method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.
result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.
New RL method nearly optimally learns policies with generative models.
problem Finding optimal policies in reinforcement learning with generative models.
method Mirror descent value iteration with KL divergence and entropy regularization.
result The method is nearly minimax-optimal for small ε \varepsilon ε -optimal policies. New algorithm reduces worst-case regret for heavy-tailed bandits.
problem Stochastic Multi-Armed Bandit problem with heavy-tailed rewards.
method Modified minimax policy MOSS with saturated empirical mean.
result Worst-case regret matching lower bound for heavy-tailed distributions.
Framework improves policy generalizability under biased training data.
problem Learning policies that generalize to a target population from biased training data.
method Characterizes sample selection bias using a selection variable, optimizes minimax value over uncertainty set, derives efficient algorithm.
result Policies generalize to target population, outperform standard methods.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
This paper explores minimax-Bayes solutions for reinforcement learning problems.
problem How to select appropriate priors for decision making under uncertainty in sequential decision making.
method Study of minimax-Bayes solutions for various reinforcement learning problems.
result Minimax policies are more robust than standard priors.
This paper optimizes off-policy evaluation in reinforcement learning with function approximation.
problem Estimating cumulative value of a new policy from logged data generated by an unknown policy.
method Regression-based fitted Q iteration method, equivalent to estimating conditional mean embedding of transition operator.
result The method is minimax-optimal, with nearly minimal estimation error.
New method for evaluating policies in complex decision-making models with hidden variables.
problem Evaluating policies in partially observable Markov decision processes with hidden confounders.
method Introduces novel identification methods and minimax estimation techniques for linking target policy's value and observed data distribution.
result Proposes three estimators for off-policy evaluation in POMDPs with latent confounders, demonstrating their effectiveness through nonasymptotic and asymptotic analysis.
Study nonparametric estimator for Markov chain transition matrices in offline setting.
problem Estimating transition matrices of finite controlled Markov chains from logged data.
method Developed sample complexity bounds and conditions for minimaxity.
result Achieving certain statistical risk requires balancing mixing properties and sample size.
New algorithm reduces RL policy optimization gap.
problem Insufficient theoretical understanding of policy optimization methods.
method Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT)
result Achieves nearly minimax optimal policy-based algorithm for tabular RL.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
Paper tackles RL with continuous actions and unmeasured confounders.
problem Offline policy learning with continuous actions and unmeasured confounders.
method Developed a novel identification result and a minimax estimator for nonparametric policy value estimation.
result Introduced a policy-gradient-based algorithm to identify the optimal policy.
New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.
problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.
method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with L 2 L^2 L 2 -convergence guarantees. result PAC guarantees for offline RL under partial coverage and realizability conditions.
Study on Q Q Q -function estimation for continuous state-action MDPs, deriving rates and conditions.
problem Estimating Q Q Q -function in off-policy evaluation for continuous state-action Markov decision processes. method Reformulated as nonparametric instrumental variables (NPIV) problem, derived minimax lower bounds, proposed sieve two-stage least squares estimator.
result First minimax lower bounds for Q Q Q -function and its derivatives in sup-norm and L 2 L^2 L 2 -norm, same as classical nonparametric regression. The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
Trust-aware MAB improves learning performance by accounting for human deviation.
problem Learning performance suffers when humans deviate from recommended policies due to lack of trust.
method Integrates a dynamic trust model into MAB framework, establishing minimax regret and proposing a two-stage trust-aware procedure.
result Proves near-optimal statistical guarantees for trust-aware MAB algorithms.
Bayesian methods suffer from the problem of how to specify prior beliefs. One interesting idea is to consider worst-case priors. This requires solving a stochastic zero-sum game. In this paper, we extend well-known results from bandit theory in order to discover minimax-Bayes policies and discuss when they are practica…
The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration. Existing algorithms provide little information about the quality of their current policy before executing it, and thus have limited use in high-stakes applications like healthcare. We address this lack of …
Study minimax rates for online learning with time-varying dynamics.
problem Online learning with time-varying state and cost dynamics.
method Non-constructive upper and lower bounds, complexity and stability terms.
result Characterization of minimax rates and necessary conditions for learnability.
The paper proposes a new policy for optimal treatment allocation based on quantile treatment effects.
problem Optimal treatment allocation policies that target distributional welfare, especially when individuals are heterogeneous.
method The approach involves allocating treatments based on the conditional quantile of individual treatment effects (QoTE), considering both prudent and negligent policymakers.
result The proposed minimax policies are robust to model uncertainty and can be generalized to various settings.
We propose a robust regression approach to off-policy evaluation (OPE) for contextual bandits. We frame OPE as a covariate-shift problem and leverage modern robust regression tools. Ours is a general approach that can be used to augment any existing OPE method that utilizes the direct method. When augmenting doubly rob…
SAMPLR optimizes for ground truth in aleatoric parameters to avoid curriculum-induced covariate shift.
problem Curriculum learning shifts training distribution, leading to suboptimal policies in aleatoric settings.
method SAMPLR optimizes ground-truth utility function, avoiding curriculum-induced covariate shift.
result SAMPLR preserves optimality under ground-truth distribution, promoting robustness across various environments.
The paper sets bounds on how much regret is unavoidable in adaptive LQR with unknown B-matrix.
problem Understanding the limits of adaptive LQR with unknown B-matrix.
method Local asymptotic minimax regret lower bounds using van Trees' inequality and Bellman error representation.
result Logarithmic regret is impossible if the parametrization induces an uninformative optimal policy.
New algorithm achieves optimal regret in average reward MDPs without prior bias information.
problem Achieving optimal regret in average reward MDPs with computational efficiency and without prior bias information.
method Projective Mitigated Extended Value Iteration (PMEVI) to compute bias-constrained optimal policies efficiently.
result First tractable algorithm with minimax optimal regret of O ~ ( s p ( h ∗ ) S A T ) \widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T}) O ( sp ( h ∗ ) S A T ) . New method learns optimal policies in presence of unmeasured confounders.
problem Optimal policy learning with unobserved confounders.
method Causal-assisted policy learning methods using instrumental variables and negative controls.
result Policies are i l d e O ( n − 1 / 2 ) ilde{\mathscr{O}}(n^{-1/2}) i l d e O ( n − 1/2 ) quantile-optimal under mild coverage assumptions. New algorithm finds optimal policies without knowing reward functions.
problem Reward-agnostic exploration in reinforcement learning.
method Designs an algorithm that explores without reward information, achieving minimax optimality.
result Achieves provable minimax optimality in finding optimal policies for multiple reward functions.
This work considers the sample and computational complexity of obtaining an ε ε ε -optimal policy in a discounted Markov Decision Process (MDP), given only access to a generative model. In this work, we study the effectiveness of the most natural plug-in approach to model-based planning: we build the maximum likelihood es…
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
Designs a single policy for collecting data to train near-optimal policies.
problem Engineering overhead in deploying minimax procedures for stochastic linear contextual bandits.
method Designs a single stochastic policy to collect data from which a near-optimal policy can be extracted.
result The designed policy can collect data from which a near-optimal policy can be extracted.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
New method for off-policy evaluation in POMDPs using future-dependent value functions.
problem Curse of horizon in off-policy evaluation for POMDPs.
method Develops future-dependent value functions and minimax learning method.
result PAC result and Bellman completeness for the proposed OPE estimator.
Algorithm extsc{Pedel} learns near-optimal policies efficiently on specific problems.
problem Learning near-optimal policies in linear MDPs with minimal samples.
method Online experiment design to focus exploration on relevant directions.
result Achieves instance-dependent complexity, outperforming minimax-optimal algorithms.
The paper addresses decision making with partially calibrated forecasts, offering a robust approach.
problem Developing a decision-making strategy for forecasts that are only partially calibrated.
method A minimax approach to mapping predictions to actions, considering worst-case distributions.
result The minimax optimal decision rule is to trust predictions and act accordingly, even for partially calibrated forecasts.
Minimax PAC bounds for learning in exogenous contextual MDPs
problem PAC learning in tabular discounted Markov decision processes with exogenous i.i.d. contexts
method Variance-reduced algorithm for policy evaluation, best-value estimation, and best-policy extraction
result Minimax optimal sample complexity
The paper tackles optimal policy learning with asymmetric counterfactual utilities in healthcare decisions.
problem Learning optimal policies from observed data with asymmetric counterfactual utilities.
method The approach involves identifying and minimizing the maximum expected utility loss using statistical decision theory and solving intermediate classification problems.
result One can learn minimax loss decision rules from observed data.
New algorithm reduces RL complexity with low switching costs.
problem Exploration-exploitation dilemma in RL with complex models.
method Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation.
result Achieves minimax optimal regret of O ( d H K ) O(d\sqrt{HK}) O ( d H K ) and near-optimal policy switching cost. Develops a minimax optimal estimator for system stability under distribution shift.
problem Ensuring system reliability under changes in the underlying environment.
method Minimax optimal estimation of stability defined in terms of acceptable performance degradation.
result Characterizes the minimax convergence rate and demonstrates practical utility.
We study a non-parametric multi-armed bandit problem with stochastic covariates, where a key complexity driver is the smoothness of payoff functions with respect to covariates. Previous studies have focused on deriving minimax-optimal algorithms in cases where it is a priori known how smooth the payoff functions are. I…
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.
Efficient algorithms for low-rank bandits using subspace recovery.
problem Contextual bandits with low-rank reward matrices.
method Spectral methods for subspace recovery, reformulating as linear bandits.
result Nearly optimal policy evaluation and best policy identification, minimax guarantees for regret minimization.
Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.
problem Estimating the distribution of returns in MDPs with high variance and bias.
method Developed a doubly robust (DR) estimator for the CDF of returns in MDPs, incorporating model-based estimation to mitigate variance issues.
result The DR estimator achieves lower variance and bias compared to IS estimators, and matches minimax lower bounds.
New algorithm reduces offline RL sample complexity for MDPs.
problem Learning optimal policies from offline data in unknown MDPs.
method Adaptive Pessimistic Value Iteration (APVI) algorithm.
result Suboptimality bound nearly matches theoretical limits.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.