Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
This tutorial reviews offline RL methods and challenges in deep learning.
problem Extracting optimal policies from large datasets without online data.
method Review of existing offline RL algorithms and potential solutions.
result Challenges and open problems in modern deep RL methods.
New algorithms improve decision-making with limited offline data.
problem Using limited offline data to cluster users for better decision-making.
method Proposed two algorithms: Off-C2LUB and Off-CLUB to address data insufficiency.
result Both algorithms outperform existing methods under limited offline user data.
A fundamental question for companies with large amount of logged data is: How to use such logged data together with incoming streaming data to make good decisions? Many companies currently make decisions via online A/B tests, but wrong decisions during testing hurt users' experiences and cause irreversible damage. A ty…
New framework converts offline to online estimation using black-box offline estimators.
problem Convert offline estimation algorithms to online estimation algorithms.
method Oracle-Efficient Online Estimation (OEOE) framework.
result Achieves near-optimal online estimation error via black-box offline estimators.
New algorithms use offline data to improve online decision-making with latent states.
problem Accelerating online sequential decision-making with latent states in offline data.
method Design end-to-end latent bandit algorithms for linear latent contextual bandits, learning latent subspace offline and using it online.
result Proves minimax optimal regret guarantees for online algorithms and practical efficiency.
Transformers learn to make decisions in new contexts from offline data.
problem Understanding when and how transformers can perform in-context reinforcement learning.
method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.
result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.
Framework reduces contextual bandit learning to offline regression with near-optimal regret.
problem Efficient learning with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that minimizes regret with near-optimal oracle calls.
result Near-optimal regret for contextual bandits with large action spaces and O ( l o g ( T ) ) O(log(T)) O ( l o g ( T )) offline oracle calls. In data-limited settings, stochastic policies can outperform deterministic ones in bandit problems.
problem Making reliable decisions with limited data in bandit problems.
method Designing TRUST, an algorithm that uses localization laws and relative pessimism.
result TRUST achieves comparable sample complexity to LCB on minimax problems but is significantly lower on few-sample problems.
OE2D framework reduces contextual bandits to offline regression for near-optimal regret.
problem Efficiently learning contextual bandits with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that reduces contextual bandits to offline regression.
result Near-optimal regret for contextual bandits with large action spaces and O ( log T ) O(\log T) O ( log T ) calls to an offline regression oracle. Meta-KeL learns kernels from offline data to improve sequential decision-making.
problem Adaptive confidence sets for prediction functions in sequential decision-making tasks.
method Meta-KeL: meta-learning a kernel from offline data; structured sparsity estimator for unknown kernel combinations.
result Valid confidence sets that become as tight as those given the true unknown kernel with increasing offline data.
Anchor-TS uses median anchoring to improve online decision-making from offline data with distribution shift.
problem Improving online decision-making from offline data with distribution shift.
method Sample-Mean Anchored Thompson Sampling (Anchor-TS) with median anchoring.
result Anchor-TS safely leverages offline data to accelerate online learning and reduces regret.
New offline RL study shows exponential sample requirement for accurate policy evaluation.
problem Understanding statistical limits of offline RL with linear function approximation.
method Analyzes necessary representational and distributional conditions for sample-efficient offline reinforcement learning.
result Even with realizability and good feature coverage, offline RL requires exponential samples for accurate policy evaluation.
Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
The paper addresses contextual optimization problems with feedback, aiming to minimize regret.
problem Contextual optimization with feedback information.
method Characterizing the optimal minimax policy in offline setting and leveraging geometric characterization in online setting to optimize cumulative regret.
result Developed an algorithm yielding logarithmic regret bound in the online setting.
Paper tackles robust offline RL for non-Markovian processes, improving efficiency and applicability.
problem Learning robust policies for non-Markovian decision processes with limited offline data.
method Proposes a novel algorithm with dataset distillation and LCB design for robust values, derived new dual forms, and introduces concentrability coefficients.
result Proves polynomial sample efficiency for finding ε-optimal robust policies.
RORL improves offline RL robustness with conservative smoothing.
problem Distribution shift and robustness issues in offline RL.
method RORL introduces regularization and conservative smoothing for robustness.
result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.
Interpole learns transparent decision-making policies from data.
problem Understanding human decision-making in opaque environments.
method Interpole combines belief-update and belief-action mapping estimation.
result Interpole provides interpretable models of decision-making behavior.
The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
New algorithm minimizes Bayesian regret in offline linear bandits.
problem Minimizing Bayesian regret in offline linear bandits.
method Proposes a new algorithm that directly minimizes upper bounds on Bayesian regret using conic optimization.
result Upper bounds are tight and guarantee superior performance compared to LCB.
An online decision-making algorithm using stochastic gradient descent for big data.
problem Efficiently updating decision rules in online decision making with big data.
method Stochastic gradient descent for online updates, asymptotic normality of estimators.
result Asymptotic normality of parameter and value estimators, enabling statistical inference.
Study risk-sensitive RL in offline settings, improving efficiency and accuracy.
problem Efficiently derive near-optimal policies for risk-sensitive RL using offline data.
method Introduced two provably sample-efficient algorithms for risk-sensitive offline RL in linear MDPs.
result First provably efficient risk-sensitive offline RL algorithms.
A framework identifies worst-case decision points in safety-critical scenarios, improving risk assessment by 10 hours.
problem Identifying worst-case outcomes in safety-critical decision-making under uncertainty.
method Explicitly estimating distributions of expected return to identify dead-ends, tuning based on risk tolerance.
result Significantly improves risk assessment, providing indications 10 hours earlier and increasing detection by 20%.
New approach uses autoregressive models to explore and quantify uncertainty in decision-making.
problem Quantifying and exploring uncertainty in online decision-making.
method Reformulates uncertainty as missing future outcomes, training autoregressive models for next-outcome prediction.
result Establishes a reduction from online learning to offline next-outcome prediction, controlling Bayesian regret by sequence prediction loss.
Bridges uplift modeling and sequential decision-making with online budget allocation.
problem Treatment allocation under budget constraints in digital advertising.
method Budget-Constrained Causal Bandits (BCCB) integrates learning, exploration, and budget pacing.
result Data-efficiency crossover: BCCB operates effectively from the first user, 3-5x lower performance variance.
Interactive IL beats BC by state-wise annotation cost.
problem Behavior Cloning struggles with annotation cost in sequential decision making.
method Proved Stagger and Warm Stagger algorithms to outperform BC.
result Interactive and hybrid IL methods outperform BC with state-wise annotation.
New algorithm reduces offline RL sample complexity for MDPs.
problem Learning optimal policies from offline data in unknown MDPs.
method Adaptive Pessimistic Value Iteration (APVI) algorithm.
result Suboptimality bound nearly matches theoretical limits.
Optimizes mobile notifications for multiple objectives using reinforcement learning.
problem Optimizing mobile notification systems for multiple objectives.
method End-to-end offline reinforcement learning with Double Deep Q-network and Conservative Q-learning.
result Demonstrates improved performance and benefits of the proposed approach.
A new Thompson Sampling framework handles uncertainty by imputing missing data.
problem Handling uncertainty in contextual bandit problems.
method Generative model to impute missing outcomes, fit policy, and select actions.
result Established a state-of-the-art regret bound that depends on generative model quality.
Proposes a method to learn from historical data for personalized decision-making.
problem Sample hunger in sequential decision-making algorithms for personalized medicine.
method Identifiable latent bandit framework using nonlinear independent component analysis.
result Optimal decision-making with shorter exploration time than classical bandits.
Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.
problem Challenges in offline reinforcement learning with deep neural networks and data dependence.
method Establishes estimation error for pessimistic offline RL using neural network approximation with C \mathcal{C} C -mixing data. result Explicit efficiency of deep adversarial offline RL frameworks demonstrated with two converging error components.
Paper tackles robust decision-making from multiple sites with shared structure.
problem Learning robust sequential decisions from heterogeneous multi-site datasets.
method Group-Robust MDPs with d-rectangular uncertainty sets, feature-wise worst-case aggregation, and cluster-level pooling.
result Proves suboptimality bound for robust planning policy under robust partial coverage assumption.
Pessimistic RL algorithm improves offline RL performance.
problem Insufficient dataset coverage in offline RL.
method Proposes a pessimistic variant of value iteration (PEVI) with a penalty function.
result Establishes upper bound on suboptimality for general MDPs, matching lower bound.
Paper tackles robust reinforcement learning with minimal data.
problem Learning robust policies from limited data in uncertain environments.
method Distributionally robust formulation, model-based algorithm combining value iteration and pessimism.
result Proves near-optimal sample complexity for robust offline RL.
Guarantees for third-person imitation learning from offline data.
problem Improving generalizability in imitation learning.
method Problem-dependent statistical learning guarantees for third-person imitation from offline observation.
result Strong performance guarantees for transferred policies in the offline setting.
Paper proposes federated offline RL for personalized medicine.
problem Privacy constraints and heterogeneity in healthcare data.
method Multi-site Markov decision process model and first federated policy optimization algorithm.
result The proposed algorithm achieves comparable suboptimality to centralized RL.
Develops a support-aware framework for reserve-policy selection in advertising markets.
problem Log-based reserve-price evaluation risks weak support and subgroup harm.
method Support-aware offline decision framework converting logged evidence into certified policies.
result Preserves the best gate-passing policy while eliminating only policies with certified regret.
Study optimal pricing and inventory control in dynamic settings with censored demand.
problem Optimal pricing and inventory control in dynamic settings with censored demand.
method Approximate optimal policy via high-order MDP, propose novel algorithms for solving Bellman equations.
result Established finite-sample regret bounds and demonstrated efficacy through numerical experiments.
Paper tackles sample-efficient offline RL, proposing data diversity and unified algorithms.
problem Sample-efficient learning from historical data for sequential decision-making.
method Proposes data diversity and unifies three offline RL algorithm classes: VS, RO, and PS.
result Comparable sample efficiency for VS, RO, and PS algorithms under standard assumptions.
Efficient offline reinforcement learning with neural networks using differentiable function approximation.
problem Statistical efficiency of offline reinforcement learning with function approximators.
method Pessimistic fitted Q-learning (PFQL) and differentiable function approximation.
result Provably efficient offline reinforcement learning with differentiable function approximation.
Paper analyzes faster convergence rates for reinforcement learning from offline data.
problem Analyzing faster convergence rates for reinforcement learning from offline data.
method Fine analysis of reinforcement learning from offline data, providing fast rates for regret convergence.
result The paper provides fast rates for the regret convergence, showing that the level of exponentiation depends on the noise in the decision-making problem.
New method improves decision-making accuracy without complex calculations.
problem Improving decision-making accuracy in machine learning.
method Introducing a new measure called calibration decision loss ( C D L K \mathsf{CDL}_K CDL K ) for structured families of post-processing functions. result Proves upper and lower bounds for natural classes K K K of post-processing functions. New algorithm reduces online decision-making regret with efficient LP re-solving and parallel first-order method.
problem Worse regret guarantees and high computational cost of LP-based OLP algorithms.
method Combines LP-based and first-order OLP methods, re-solving LP subproblems periodically and using parallel first-order method.
result Achieves O ( log ( T / f ) + f ) \mathscr{O}(\log (T/f) + \sqrt{f}) O ( log ( T / f ) + f ) regret, balancing computational efficiency and superior regret guarantee. Pre-trained LLM adapted with LoRA improves offline RL for quantitative trading.
problem Challenges in offline RL for quantitative trading due to complex temporal dependencies and overfitting.
method Integrates pre-trained GPT-2 weights and LoRA for efficient fine-tuning of a Decision Transformer.
result Outperforms existing offline RL methods in certain trading scenarios.
Unified LP framework for offline reward learning from human demonstrations and feedback.
problem Reward learning from human demonstrations and feedback with robustness and sample efficiency.
method A novel linear programming framework for offline reward learning.
result Unified LP framework achieves better performance compared to MLE.
Paper analyzes sample complexity of offline MABs with KL regularization.
problem Optimizing sample complexity for offline decision-making with KL-regularized metrics.
method Sharp analysis of KL-PCB, providing upper and lower bounds.
result Characterizes sample complexity for offline MABs with KL regularization.
We consider the problem of sequentially making decisions that are rewarded by "successes" and "failures" which can be predicted through an unknown relationship that depends on a partially controllable vector of attributes for each instance. The learner takes an active role in selecting samples from the instance pool. T…