The paper shows how to learn near-optimal behavior in reinforcement learning with rich observations.
problem Learning near-optimal behavior in reinforcement learning with rich observations and function approximation.
method Introduces a new model called contextual decision processes and a new algorithm that engages in systematic exploration to learn these processes with low Bellman rank.
result The algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters.
A new algorithm for cryo-EM data collection that balances reward and latency.
problem Optimizing data collection in cryo-EM experiments with action delays.
method Latency-aware contextual bandit framework and COAF algorithm.
result The COAF algorithm efficiently maximizes reward over time in cryo-EM experiments.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
A new algorithm uses concavity in Gaussian processes to optimize decisions in bandit problems.
problem Optimizing decisions in sequential problems with context-dependent rewards.
method Proposes a UCB algorithm using a shape-constrained reward function estimator based on a Gaussian Process model with concavity constraints.
result Derives regret bounds for the proposed UCB algorithm.
Unified framework for human-like decision making in various sequential tasks.
problem Real-life decision-making involves diverse strategies leading to similar outcomes.
method Two-stream reward processing mechanism for flexible and unified models.
result Framework unified MAB, CB, and RL with comparable performance.
New algorithm for partially observable contexts in finance.
problem Decision making based on partially observable, correlated market information.
method EMKF-Bandit algorithm integrating system identification, filtering, and bandit algorithms.
result Sub-linear regret under conditions on filtering.
CB-RL solves complex decision-making problems with contextual information and exogenous events.
problem Optimal policy in strategic decision-making problems that depend on environmental configuration and exogenous events.
method Contextual Bilevel Reinforcement Learning (CB-RL) with a stochastic Hyper Policy Gradient Descent (HPGD) algorithm.
result Demonstrated convergence and performance of the HPGD algorithm for reward shaping and tax design.
Simple method improves exploration in various decision problems.
problem Improving exploration in sequential decision problems.
method Parameterized Exploration (PE) method that considers time horizon and state of knowledge.
result PE outperforms un-tuned methods in various bandit and decision problem settings.
Novel algorithm reduces feature inclusion in online decision-making.
problem Optimizing decision-making for personalized user experiences with fairness.
method Online Batched Sequential Inclusion (OBSI) algorithm for sequential feature inclusion.
result OBSI outperforms other algorithms in terms of regret, relevance of features, and compute.
Unified framework for sequential decision making using meta-learning surrogate models.
problem Sequential decision making problems in various domains.
method Probabilistic model-based approach with meta-learning for data-efficient adaptation.
result Efficient and general black-box learning approach across different problem domains.
Novel framework for contextual anomaly detection models uncertainty.
problem Identifying anomalies in target variables influenced by contextual variables.
method Normalcy score (NS) framework using heteroscedastic Gaussian process regression.
result NS outperforms state-of-the-art methods in detection accuracy and interpretability.
New algorithm matches best regret bound for tabular Contextual Bandit problems.
problem Learning from observations in uncertain environments.
method A minor variant of a reinforcement learning algorithm for MDPs.
result The algorithm matches the best possible regret bound i l d e O ( S A T ) ilde O (\sqrt{SAT}) i l d e O ( S A T ) for tabular Contextual Bandit problems. Method learns evolving policies in healthcare contexts.
problem Understanding non-stationary behavior in evolving decision-making processes.
method Inverse Contextual Bandits (ICB) approach for learning interpretable representations of non-stationary behavior.
result Demonstrated applicability and accuracy of ICB method in liver transplantation policies.
CPR models complex decision processes by breaking them into context-specific policies, improving interpretability and accuracy.
problem Interpreting dynamic human decision-making processes in medical contexts.
method Develops Contextualized Policy Recovery (CPR) framework for multi-task learning, modeling each context-specific policy as a linear map.
result Achieves state-of-the-art performance in predicting medical decisions, closing the gap between interpretable and black-box methods.
The paper develops a reinforcement learning model to estimate ad impact considering delayed and cumulative effects.
problem Accurately estimating ad impact considering delayed and long-term effects, cumulative impacts, and customer heterogeneity.
method Modeling ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards, proposing a two-stage maximum likelihood estimator and reinforcement learning algorithm.
result Achieves a near-optimal regret bound of O ~ ( d H 2 T ) \tilde{O}{(dH^2\sqrt{T})} O ~ ( d H 2 T ) , validating the approach through simulation experiments. Efficient algorithm for learning from indirect feedback in complex decision-making scenarios.
problem Learning from indirect feedback in realistic scenarios with personalized mechanisms.
method IGW algorithm for policy optimization, extending reward-estimator construction from single-step to multi-step.
result Achieves sublinear regret guarantee for contextual episodic MDPs with personalized feedback.
New RL algorithms learn from context in MDPs with smooth dynamics.
problem Learning in MDPs with varying dynamics based on context.
method Proposes algorithms for CMDPs with smooth context-MDP mappings.
result PAC learning algorithm for linear CMDPs.
A new method for contextual bandits using decision trees.
problem Applying efficient algorithms for contextual bandits in practice requires domain expertise.
method Uses decision trees to model context-reward relationships and a bootstrapping approach for exploration-exploitation.
result Demonstrates improved performance on various datasets compared to existing methods.
New algorithm reduces regret in RL with adversarial corruption.
problem Adversarial corruption in reinforcement learning.
method Uncertainty-weighted least-squares regression and weighted uncertainty estimator.
result Achieves regret of i l d e O ( T + ζ ) ilde{O}(\sqrt{T} + ζ) i l d e O ( T + ζ ) for contextual bandits. We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a cu…
Paper proposes a risk-aware decision-making framework for real-world sequential decisions.
problem Real-world sequential decision-making problems often have critical constraints that learning solutions often neglect.
method Actor multi-critic architecture with risk characterization.
result Our approach consistently satisfies system constraints with minimal performance toll.
Paper tackles online learning in large MDPs with low Bellman rank using AVE algorithm.
problem Online learning of MDPs with large state spaces.
method Develops AVE algorithm inspired by OLIVE, using contextual bandit problems and elimination steps.
result Achieves n \sqrt{n} n -regret for learning optimal value function in MDPs with function approximation and low Bellman rank. Human irrationality can improve AI design, study shows.
problem Improving AI by learning from human decision-making biases.
method Developed a novel POMDP model to simulate human decision-making in contextual choice tasks.
result Reinforcement learners can exploit human irrationalities to make better decisions.
New framework handles dynamic contexts in reinforcement learning.
problem Learning in environments where contexts change over time.
method Dynamic Contextual Markov Decision Processes (DCMDPs) with logistic aggregation.
result Upper-confidence-bound style algorithm with regret bounds.
New RL algorithms show model-based methods are more efficient than model-free ones in complex decision processes.
problem Efficient reinforcement learning in contextual decision processes with strategic exploration.
method Design of new model-based RL algorithms with sample complexity governed by witness rank.
result Exponential separation between model-based and model-free RL in some rich-observation settings.
Framework reduces contextual bandit learning to offline regression with near-optimal regret.
problem Efficient learning with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that minimizes regret with near-optimal oracle calls.
result Near-optimal regret for contextual bandits with large action spaces and O ( l o g ( T ) ) O(log(T)) O ( l o g ( T )) offline oracle calls. Aims to improve personalized treatment decisions through Bayesian experimental design.
problem Evaluating and improving personalized treatment decisions in contexts like customer service.
method Model-agnostic Bayesian Experimental Design to efficiently gather data and avoid highly sub-optimal treatments.
result Our method achieves superior performance in evaluating and improving treatment decisions compared to traditional approaches.
The paper proposes a method to learn and leverage contextual preference distributions for better decision-making.
problem Heterogeneous and context-dependent human preferences in decision-making problems.
method A sequential learning-and-optimization pipeline using a bounded-variance score function gradient estimator to train a predictive model mapping contextual features to preference distributions.
result The approach reduces average post-decision surprise by up to 25 times compared to risk-averse baselines in a ridesharing environment.
A new method for decision tree selection in recommendation systems.
problem Feature-based selection of a single tree from an ensemble for dynamic interpretation.
method A multi-armed contextual bandit recommendation framework that trains a system on top of Random Forests to identify the most relevant tree.
result The dynamic method outperforms an independent CART tree and is comparable to Random Forest in predictive performance.
OE2D framework reduces contextual bandits to offline regression for near-optimal regret.
problem Efficiently learning contextual bandits with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that reduces contextual bandits to offline regression.
result Near-optimal regret for contextual bandits with large action spaces and O ( log T ) O(\log T) O ( log T ) calls to an offline regression oracle. New method for contextual bandit with missing rewards.
problem Contextual bandit with missing rewards in online settings.
method Combining contextual bandit approach with unsupervised learning (clustering) to estimate missing rewards.
result Promising empirical results on real-life datasets.
New method for contextual bandits with corrupted context.
problem Contextual bandits with corrupted context in online settings.
method Combining contextual bandit and multi-armed bandit approaches.
result Improved learning from all iterations, including corrupted ones.
The paper addresses statistical inference for online decision-making in a contextual bandit setting.
problem Understanding the performance of reward models in online decision-making with contextual information.
method The paper uses the contextual bandit framework with a linear reward model and the ε \varepsilon ε -greedy policy to address the exploration-exploitation dilemma. It employs the martingale central limit theorem and inverse propensity score weighting to establish asymptotic normality of parameter estimators. result The online ordinary least squares estimator and the online weighted least squares estimator are asymptotically normal, providing insights into the performance of the reward model.
Universal algorithm learns unknown distribution for various decision-making problems.
problem Various statistical measures in contextual sequential decision-making.
method Infinite-dimensional functional regression oracle for cumulative distribution functions.
result Utility regret rate bounded by polynomial decay of eigenvalue sequence.
New algorithm estimates treatment effects for more efficient contextual bandits.
problem Contextual bandits struggle with action-independent reward redundancies.
method Reduces contextual bandits to heterogeneous treatment effect estimation.
result Heterogeneous treatment effect estimation leads to more efficient model estimation.
No-regret algorithm for contextual RL with GLM mappings.
problem Learning near-optimal policies in episodic MDPs with contextual information.
method Proposes no-regret online RL algorithm using optimistic and randomized exploration methods.
result Improves previous bounds and provides a lower bound for the setting.
Combines machine learning and optimization for real-time decision-making.
problem Optimizing decisions in contextually constrained problems.
method Generative model combining interior point methods and adversarial learning.
result Generative model produces optimal decisions with in-sample and out-of-sample guarantees.
Study wSAA for contextual decisions, improving uncertainty quantification under computational constraints.
problem Uncertainty quantification limitations in wSAA for contextual stochastic optimization.
method Establish central limit theorems and asymptotic-normality-based confidence intervals for optimal costs.
result Over-optimizing can mitigate misspecification and preserve asymptotic normality, albeit at a slower convergence rate.
This study analyzes how RNNs process context in sentiment analysis.
problem Understanding how recurrent neural networks process context in sentiment analysis.
method Developed methods to reverse engineer RNNs, identifying contextual effects and quantifying their strength and timescale.
result Identified inputs that induce contextual effects and quantified their properties.
The paper addresses contextual optimization problems with feedback, aiming to minimize regret.
problem Contextual optimization with feedback information.
method Characterizing the optimal minimax policy in offline setting and leveraging geometric characterization in online setting to optimize cumulative regret.
result Developed an algorithm yielding logarithmic regret bound in the online setting.
New method for online statistical inference in contextual bandits using SGD.
problem Online decision-making in contextual bandits with statistical inference.
method Weighted stochastic gradient descent for adaptive data collection.
result Asymptotic normality of the parameter estimator with improved efficiency.
Proposes an online model for LLM cascading with adaptive API selection.
problem Adaptive querying and selection of LLM APIs in a context-dependent environment.
method Develops a learning approach combining GMM estimation and UCB-style bounds.
result Achieves cumulative regret of O ~ ( T ) \widetilde O(\sqrt T) O ( T ) over T T T periods. The paper presents a dataset and evaluates context-aware TV content recommendations.
problem Lack of data supporting context-aware TV content recommendation systems.
method Developed a dataset of TV consumption with contextual information and evaluated prediction performance.
result Including contextual features improves prediction accuracy, with social and temporal context contributing significantly.
Algorithm minimizes regret in dueling bandits with contextualized utilities.
problem Minimizing regret in dueling bandits with context-dependent utilities.
method Proposes CoLSTIM algorithm based on perturbed utility estimates.
result Achieves regret of order i l d e O ( d T ) ilde O(\sqrt{dT}) i l d e O ( d T ) . SPARKLE handles high-dimensional covariates for online decision-making.
problem Complex reward-covariate relationships in high-dimensional settings.
method SPARKLE uses a sparse additive reward model with doubly penalized estimator and adaptive screening.
result SPARKLE achieves sublinear regret bound logarithmic in covariate dimensionality.
Efficiently handles contextual bandits with diffusion models.
problem Challenges in online decision-making with contextual bandits.
method Leverage pre-trained diffusion models as priors to capture action dependencies.
result Developed an algorithm for efficient posterior approximation.
AACC improves RL performance in changing environments.
problem Deterioration of RL performance in real-world tasks.
method Formalizes CMDPs, proposes AACC for contextual RL.
result AACC outperforms baselines in various simulated environments.
Improved algorithm for misspecified MLMDPs with bounded regret and space/time complexities.
problem Misspecified linear Markov decision processes.
method Proposes an algorithm with three desirable properties: bounded regret, bounded space/time complexities, and no need for misspecification input.
result Regret scales as K max { ε e x t m i s , ε e x t t o l } K \max \{ \varepsilon_{ ext{mis}}, \varepsilon_{ ext{tol}} \} K max { ε e x t mi s , ε e x t t o l } , improving existing bounds.