New RL method nearly optimally learns policies with generative models.
problem Finding optimal policies in reinforcement learning with generative models.
method Mirror descent value iteration with KL divergence and entropy regularization.
result The method is nearly minimax-optimal for small ε \varepsilon ε -optimal policies. Study minimax off-policy evaluation in multi-armed bandits with known and unknown behavior policies.
problem Evaluate policies in multi-armed bandits with unknown behavior policies.
method Develop minimax rate-optimal procedures for known and unknown behavior policies, including the Switch estimator and Chebyshev polynomial-based estimator.
result Plug-in estimator achieves optimal competitive ratio up to a logarithmic factor when behavior policy is unknown.
Unified minimax value interval for off-policy evaluation and optimization.
problem Overcoming the exponential variance in off-policy evaluation and policy optimization.
method Unified minimax value interval using marginalized importance weights.
result Unified value interval with double robustness, valid when either value-function or importance-weight class is well specified.
New algorithm reduces RL policy optimization gap.
problem Insufficient theoretical understanding of policy optimization methods.
method Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT)
result Achieves nearly minimax optimal policy-based algorithm for tabular RL.
Improves RL generalization by minimizing adversarial risk.
problem Overfitting to training environments and poor generalization to unseen scenarios.
method Introduces minimax formulation and distributional framework to RL.
result Trained policy shows improved generalization to different environments.
The paper tackles robust policy learning from multiple data sources.
problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.
method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.
result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
This paper optimizes off-policy evaluation in reinforcement learning with function approximation.
problem Estimating cumulative value of a new policy from logged data generated by an unknown policy.
method Regression-based fitted Q iteration method, equivalent to estimating conditional mean embedding of transition operator.
result The method is minimax-optimal, with nearly minimal estimation error.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
Framework improves policy generalizability under biased training data.
problem Learning policies that generalize to a target population from biased training data.
method Characterizes sample selection bias using a selection variable, optimizes minimax value over uncertainty set, derives efficient algorithm.
result Policies generalize to target population, outperform standard methods.
Paper tackles RL with continuous actions and unmeasured confounders.
problem Offline policy learning with continuous actions and unmeasured confounders.
method Developed a novel identification result and a minimax estimator for nonparametric policy value estimation.
result Introduced a policy-gradient-based algorithm to identify the optimal policy.
New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.
problem Offline RL with limited exploration and assumptions about data coverage and Q-function realizability.
method Proposes minimax learning algorithms to estimate soft or vanilla Q-functions with L 2 L^2 L 2 -convergence guarantees. result PAC guarantees for offline RL under partial coverage and realizability conditions.
Optimizes treatment allocation using covariates for better outcomes.
problem Improving treatment allocation in multi-armed bandit problems.
method Maximizes a functional of the conditional potential outcome distribution.
result Developed expected regret lower bounds and near minimax optimal policy.
New algorithm achieves optimal regret in average reward MDPs without prior bias information.
problem Achieving optimal regret in average reward MDPs with computational efficiency and without prior bias information.
method Projective Mitigated Extended Value Iteration (PMEVI) to compute bias-constrained optimal policies efficiently.
result First tractable algorithm with minimax optimal regret of O ~ ( s p ( h ∗ ) S A T ) \widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T}) O ( sp ( h ∗ ) S A T ) . The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration. Existing algorithms provide little information about the quality of their current policy before executing it, and thus have limited use in high-stakes applications like healthcare. We address this lack of …
The paper sets bounds on how much regret is unavoidable in adaptive LQR with unknown B-matrix.
problem Understanding the limits of adaptive LQR with unknown B-matrix.
method Local asymptotic minimax regret lower bounds using van Trees' inequality and Bellman error representation.
result Logarithmic regret is impossible if the parametrization induces an uninformative optimal policy.
New method learns optimal policies in presence of unmeasured confounders.
problem Optimal policy learning with unobserved confounders.
method Causal-assisted policy learning methods using instrumental variables and negative controls.
result Policies are i l d e O ( n − 1 / 2 ) ilde{\mathscr{O}}(n^{-1/2}) i l d e O ( n − 1/2 ) quantile-optimal under mild coverage assumptions. SAMPLR optimizes for ground truth in aleatoric parameters to avoid curriculum-induced covariate shift.
problem Curriculum learning shifts training distribution, leading to suboptimal policies in aleatoric settings.
method SAMPLR optimizes ground-truth utility function, avoiding curriculum-induced covariate shift.
result SAMPLR preserves optimality under ground-truth distribution, promoting robustness across various environments.
Trust-aware MAB improves learning performance by accounting for human deviation.
problem Learning performance suffers when humans deviate from recommended policies due to lack of trust.
method Integrates a dynamic trust model into MAB framework, establishing minimax regret and proposing a two-stage trust-aware procedure.
result Proves near-optimal statistical guarantees for trust-aware MAB algorithms.
The paper proposes a new policy for optimal treatment allocation based on quantile treatment effects.
problem Optimal treatment allocation policies that target distributional welfare, especially when individuals are heterogeneous.
method The approach involves allocating treatments based on the conditional quantile of individual treatment effects (QoTE), considering both prudent and negligent policymakers.
result The proposed minimax policies are robust to model uncertainty and can be generalized to various settings.
Study on Q Q Q -function estimation for continuous state-action MDPs, deriving rates and conditions.
problem Estimating Q Q Q -function in off-policy evaluation for continuous state-action Markov decision processes. method Reformulated as nonparametric instrumental variables (NPIV) problem, derived minimax lower bounds, proposed sieve two-stage least squares estimator.
result First minimax lower bounds for Q Q Q -function and its derivatives in sup-norm and L 2 L^2 L 2 -norm, same as classical nonparametric regression. New algorithm finds optimal policies without knowing reward functions.
problem Reward-agnostic exploration in reinforcement learning.
method Designs an algorithm that explores without reward information, achieving minimax optimality.
result Achieves provable minimax optimality in finding optimal policies for multiple reward functions.
Algorithm extsc{Pedel} learns near-optimal policies efficiently on specific problems.
problem Learning near-optimal policies in linear MDPs with minimal samples.
method Online experiment design to focus exploration on relevant directions.
result Achieves instance-dependent complexity, outperforming minimax-optimal algorithms.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
This work considers the sample and computational complexity of obtaining an ε ε ε -optimal policy in a discounted Markov Decision Process (MDP), given only access to a generative model. In this work, we study the effectiveness of the most natural plug-in approach to model-based planning: we build the maximum likelihood es…
Designs a single policy for collecting data to train near-optimal policies.
problem Engineering overhead in deploying minimax procedures for stochastic linear contextual bandits.
method Designs a single stochastic policy to collect data from which a near-optimal policy can be extracted.
result The designed policy can collect data from which a near-optimal policy can be extracted.
New algorithm reduces RL complexity with low switching costs.
problem Exploration-exploitation dilemma in RL with complex models.
method Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation.
result Achieves minimax optimal regret of O ( d H K ) O(d\sqrt{HK}) O ( d H K ) and near-optimal policy switching cost. MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
New algorithm reduces worst-case regret for heavy-tailed bandits.
problem Stochastic Multi-Armed Bandit problem with heavy-tailed rewards.
method Modified minimax policy MOSS with saturated empirical mean.
result Worst-case regret matching lower bound for heavy-tailed distributions.
New bounds for learning near-optimal policies in CMDPs with constraints.
problem Optimizing policies in CMDPs with constraints.
method Model-based algorithm addressing relaxed and strict feasibility.
result Near-optimal sample complexity bounds for CMDPs.
New methods for off-policy evaluation in reinforcement learning.
problem Improving off-policy evaluation in reinforcement learning.
method Developed MWL and MQL estimators for importance weights and value functions.
result MWL and MQL provide a unified view of algorithms in reinforcement learning.
New algorithms find optimal policies without knowing MDP span.
problem Finding optimal policies in MDPs without knowing span.
method Horizon calibration and span penalization techniques.
result First algorithms achieving optimal span-based complexity without prior knowledge.
Develops a minimax optimal estimator for system stability under distribution shift.
problem Ensuring system reliability under changes in the underlying environment.
method Minimax optimal estimation of stability defined in terms of acceptable performance degradation.
result Characterizes the minimax convergence rate and demonstrates practical utility.
The paper addresses decision making with partially calibrated forecasts, offering a robust approach.
problem Developing a decision-making strategy for forecasts that are only partially calibrated.
method A minimax approach to mapping predictions to actions, considering worst-case distributions.
result The minimax optimal decision rule is to trust predictions and act accordingly, even for partially calibrated forecasts.
We study the global convergence of generative adversarial imitation learning for linear quadratic regulators, which is posed as minimax optimization. To address the challenges arising from non-convex-concave geometry, we analyze the alternating gradient algorithm and establish its Q-linear rate of convergence to a uniq…
PROPO tackles non-stationary MDPs with efficient policy optimization.
problem Non-stationary MDPs with varying reward and transition kernels.
method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.
result PROPO achieves near-optimal performance in non-stationary MDPs.
The paper tackles optimal policy learning with asymmetric counterfactual utilities in healthcare decisions.
problem Learning optimal policies from observed data with asymmetric counterfactual utilities.
method The approach involves identifying and minimizing the maximum expected utility loss using statistical decision theory and solving intermediate classification problems.
result One can learn minimax loss decision rules from observed data.
This paper explores minimax-Bayes solutions for reinforcement learning problems.
problem How to select appropriate priors for decision making under uncertainty in sequential decision making.
method Study of minimax-Bayes solutions for various reinforcement learning problems.
result Minimax policies are more robust than standard priors.
A single policy suffices for near-optimal parallel exploration in RL.
problem Quantitative effects of parallel exploration in reward-free RL.
method Using a single policy to guide exploration across all agents.
result Near-linear speedup and near-minimax optimality for linear MDPs.
A robust method for off-policy evaluation in contextual bandits.
problem Evaluating policies when direct methods are unavailable.
method Robust regression approach to off-policy evaluation.
result Superior empirical performance across benchmarks.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
Proposes a fairness criterion for multi-objective optimization in classification.
problem Ensuring fairness in classification models across different groups.
method Formulates a minimax Pareto fairness criterion and provides an optimization algorithm.
result Demonstrates improved fairness compared to existing methods on various real-world datasets.
New method for evaluating policies in complex decision-making models with hidden variables.
problem Evaluating policies in partially observable Markov decision processes with hidden confounders.
method Introduces novel identification methods and minimax estimation techniques for linking target policy's value and observed data distribution.
result Proposes three estimators for off-policy evaluation in POMDPs with latent confounders, demonstrating their effectiveness through nonasymptotic and asymptotic analysis.
Improved gap-dependent bounds for reinforcement learning with linear approximations.
problem Achieving nearly minimax-optimal performance with linear function approximation.
method Developed and analyzed the LSVI-UCB++ algorithm and its concurrent variant.
result First gap-dependent regret bound for nearly minimax-optimal algorithm LSVI-UCB++.
Study nonparametric estimator for Markov chain transition matrices in offline setting.
problem Estimating transition matrices of finite controlled Markov chains from logged data.
method Developed sample complexity bounds and conditions for minimaxity.
result Achieving certain statistical risk requires balancing mixing properties and sample size.
Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.
problem Non-rectangular robust Markov decision processes under average-reward criterion.
method Proves history-dependent policies are robust-optimal, introduces transient-value framework, constructs epoch-based policy.
result Existence and properties of robust optimal policies, transient value bounds.
Unified framework for risk-aware policy learning in contextual bandits.
problem Optimizing decision rules in high-stakes domains with adverse outcomes.
method Distributional framework for Lipschitz-continuous risk functionals, with novel empirical concentration inequalities.
result Data-dependent suboptimality bounds with an i l d e O ( 1 / n ) ilde{\mathcal{O}}(1/\sqrt{n}) i l d e O ( 1/ n ) rate, matching risk-neutral offline policy optimization.