Statistical inference for misspecified contextual bandits is challenging due to adaptivity issues.
problem Statistical inference for misspecified contextual bandits
method Inverse-probability-weighted Z-estimation framework
result Consistent and asymptotically normal estimator with sandwich variance estimator
New estimator reduces variance in off-policy evaluation for contextual bandits.
problem High variance in current OPE methods for contextual bandits.
method Marginal Density Ratio (MR) estimator focusing on marginal distribution shift.
result MR estimator reduces variance compared to IPW and DR methods.
Improved off-policy selection and learning in contextual bandits with better guarantees.
problem Selecting or training a reward-maximizing policy using data from a fixed behavior policy.
method A betting-based confidence bound applied to an inverse propensity weight sequence for off-policy selection, and a freezing condition for off-policy learning.
result The proposed methods achieve significantly improved guarantees over prior work, especially in small-data regimes.
The paper addresses statistical inference for online decision-making in a contextual bandit setting.
problem Understanding the performance of reward models in online decision-making with contextual information.
method The paper uses the contextual bandit framework with a linear reward model and the ε \varepsilon ε -greedy policy to address the exploration-exploitation dilemma. It employs the martingale central limit theorem and inverse propensity score weighting to establish asymptotic normality of parameter estimators. result The online ordinary least squares estimator and the online weighted least squares estimator are asymptotically normal, providing insights into the performance of the reward model.
Kernelized bandit algorithm tackles adaptive contextual bandits with single-index models.
problem Adaptive contextual bandits with single-index models and unknown link functions.
method Kernelized ε-greedy algorithm combining Stein-based index estimation and kernel ridge regression for reward functions.
result Unified framework for simultaneous learning and inference in single-index contextual bandits.
Method learns evolving policies in healthcare contexts.
problem Understanding non-stationary behavior in evolving decision-making processes.
method Inverse Contextual Bandits (ICB) approach for learning interpretable representations of non-stationary behavior.
result Demonstrated applicability and accuracy of ICB method in liver transplantation policies.
This work bridges federated learning and contextual bandits, enhancing FL's utility.
problem Limited use of federated learning in contextual bandits despite its potential.
method Proposes FedIGW, a novel federated contextual bandits design that leverages regression-based algorithms and integrates various FL components.
result FedIGW better harnesses FL innovations and provides flexible, modular, and seamless integration of FL elements.
CAEL-MIPS learns embeddings to improve MIPS for better OPE in contextual bandits.
problem High variance in IPS weighting for OPE in large action spaces.
method Context-Action Embedding Learning (CAEL) for MIPS to minimize MSE.
result CAEL-MIPS outperforms baselines in MSE for OPE in contextual bandits.
A new method combines online and offline learning to tackle contextual bandits with missing action support.
problem Learning optimal policies with logged data when the logging policy has deficient support.
method Hybrid approach using online exploration to exploit supported actions and offline learning to avoid unnecessary explorations.
result Determines an optimal policy with theoretical guarantees using minimal online explorations.
Observer learns optimal policy from learner's actions without rewards.
problem Learning optimal policy from non-rewarded actions of a non-stationary learner.
method Two-Phase Suffix Imitation framework.
result Observer achieves convergence rate of O ~ ( 1 / N ) \tilde O(1/\sqrt{N}) O ~ ( 1/ N ) . We study the off-policy evaluation problem---estimating the value of a target policy using data collected by another policy---under the contextual bandit model. We consider the general (agnostic) setting without access to a consistent model of rewards and establish a minimax lower bound on the mean squared error (MSE).…
New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
New method for online statistical inference in contextual bandits using SGD.
problem Online decision-making in contextual bandits with statistical inference.
method Weighted stochastic gradient descent for adaptive data collection.
result Asymptotic normality of the parameter estimator with improved efficiency.
Contextual bandit methods fail with deficient support data.
problem Learning from support-deficient data in contextual bandits.
method Three approaches to IPS-based learning: action space restriction, reward extrapolation, and policy space restriction.
result Systematic analysis and empirical evaluation of approaches to IPS-based learning.
We stabilize the Kumaraswamy distribution for efficient sampling and differentiation.
problem Numerical instabilities in the Kumaraswamy distribution's inverse CDF and log-pdf.
method Identified and resolved numerical issues, introduced a stabilized KS distribution.
result Stabilized Kumaraswamy distribution supports efficient sampling and differentiation.
Paper solves stochastic contextual linear bandits using linear bandit algorithms.
problem Stochastic contextual linear bandits with unknown context distribution.
method Establishes a reduction framework to convert to linear bandit problems.
result Achieves nearly optimal regret bound of O ( d T log T ) O(d\sqrt{T\log T}) O ( d T log T ) . In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of inputs with corresponding scores. When the inputs are high-dimensional and valid inputs constitute a small subset of this space (e.g., valid protein s…
Paper tackles domain adaptation for contextual bandits with sub-linear regret.
problem Adapting contextual bandit algorithms across domains with distribution shift.
method Learn a bandit model for the target domain using feedback from the source domain.
result Sub-linear regret bound maintained across domains.
New algorithm learns optimal exploration parameters for contextual bandits.
problem Learning optimal exploration in contextual bandits.
method Proposes two algorithms that learn optimal exploration parameters online based on context and reward.
result Demonstrates improved performance in learning optimal exploration compared to traditional methods.
A framework for auto-tuning hyper-parameters in contextual bandit algorithms.
problem Auto-tuning hyper-parameters in real-time for contextual bandit algorithms.
method Proposes a Syndicated Bandits framework to learn multiple hyper-parameters dynamically.
result Achieves optimal regret bounds under certain scenarios and handles multiple contextual bandit algorithms.
Develops algorithms for CCBs with non-linear costs, improving safety and performance.
problem Safety constraints in sequential decision making with non-linear arm costs.
method Innovative algorithms using Inverse Gap Weighting (IGW) and online regression oracle.
result Sub-linear regret bounds for C-SquareCB and first-order regret for C-FastCB.
Thompson Sampling bounds for contextual bandits with sub-Gaussian rewards.
problem Improving the performance of Thompson Sampling in contextual bandits with sub-Gaussian rewards.
method Proved comprehensive bounds on Thompson Sampling expected cumulative regret based on mutual information and lifted information ratio for sub-Gaussian rewards.
result Explicit regret bounds for various contextual bandit scenarios.
New method for contextual bandits with corrupted context.
problem Contextual bandits with corrupted context in online settings.
method Combining contextual bandit and multi-armed bandit approaches.
result Improved learning from all iterations, including corrupted ones.
The study explores whether model selection guarantees apply to contextual bandits.
problem Applying model selection guarantees to contextual bandits.
method Investigates whether similar guarantees for model selection in statistical learning can be extended to contextual bandit learning.
result Initial findings suggest that model selection guarantees may not directly apply to contextual bandits.
Faster algorithm reduces contextual bandit regret with fewer offline regression calls.
problem Optimizing reward in contextual bandits with unknown functions.
method Designing a simple algorithm with O ( log T ) {O}(\log T) O ( log T ) offline regression calls. result Achieves statistically optimal regret with minimal offline calls.
Efficiently handles contextual bandits with diffusion models.
problem Challenges in online decision-making with contextual bandits.
method Leverage pre-trained diffusion models as priors to capture action dependencies.
result Developed an algorithm for efficient posterior approximation.
IDS improves reinforcement learning with contextual information.
problem Optimizing IDS for contextual reinforcement learning.
method Investigated contextual bandit problems and proposed a computationally-efficient IDS.
result Contextual IDS outperforms conditional IDS by considering future contexts.
First robust bandit algorithm for contextual bandits with sub-linear regret.
problem Vulnerability of linear contextual bandit algorithms to adversarial attacks.
method Proposes a robust bandit algorithm for stochastic linear contextual bandits under fully adaptive and omniscient attacks.
result Sub-linear regret under various attacks without requiring attack information.
Proposes MDR estimator for unbiased OPE with large action spaces.
problem Severe bias and variance tradeoffs in OPE with large action spaces.
method Marginalized Doubly Robust (MDR) estimator, reducing variance and bias.
result MDR estimator is unbiased under weaker assumptions than MIPS.
New algorithm improves learning efficiency in multi-task contextual bandits.
problem Improving learning efficiency in multi-task contextual bandits.
method Alternating projected gradient descent (GD) and minimization estimator for low-rank feature matrix recovery.
result Proved regret bound for multi-task learning algorithm.
New method for contextual bandit with missing rewards.
problem Contextual bandit with missing rewards in online settings.
method Combining contextual bandit approach with unsupervised learning (clustering) to estimate missing rewards.
result Promising empirical results on real-life datasets.
New algorithm for contextual dueling bandits achieves nearly optimal regret.
problem Contextual dueling bandits with feedback on preferred options.
method Proposes FGTS.CDB, a Thompson sampling algorithm for linear contextual dueling bandits.
result Achieves nearly minimax-optimal regret of i l d e O ( d T ) ilde{\mathcal{O}}(d\sqrt T) i l d e O ( d T ) . New algorithms minimize regret in both adversarial and stochastic contexts.
problem Minimizing regret in linear contextual bandits.
method Best-of-both-worlds algorithms using FTRL with Shannon entropy regularizer.
result Achieves near-optimal regret bounds in both adversarial and stochastic regimes.
The paper tackles minimax optimality in continuum contextual bandits with Hölder continuity.
problem Minimizing regret in a continuum of contexts with Hölder continuity.
method Proves a static-to-contextual regret conversion theorem and analyzes various dependency cases.
result Achieves minimax optimal contextual regret for convex and strongly convex bandits.
New algorithm estimates treatment effects for more efficient contextual bandits.
problem Contextual bandits struggle with action-independent reward redundancies.
method Reduces contextual bandits to heterogeneous treatment effect estimation.
result Heterogeneous treatment effect estimation leads to more efficient model estimation.
Contextual bandit algorithms provide principled online learning solutions to balance the exploitation-exploration trade-off in various applications such as recommender systems. However, the learning speed of the traditional contextual bandit algorithms is often slow due to the need for extensive exploration. This poses…
We consider the stochastic linear (multi-armed) contextual bandit problem with the possibility of hidden simple multi-armed bandit structure in which the rewards are independent of the contextual information. Algorithms that are designed solely for one of the regimes are known to be sub-optimal for the alternate regime…
GLCB uses Gated Linear Networks for online contextual bandits.
problem Online learning in contextual bandits with uncertainty estimation.
method Gated Linear Networks (GLNs) for prediction and uncertainty estimation.
result GLCB outperforms state-of-the-art methods in online contextual bandits.
LMC-TS uses MCMC for efficient posterior sampling in contextual bandits.
problem Efficiency of Thompson sampling for high-dimensional contextual bandits.
method Langevin Monte Carlo for direct posterior sampling.
result LMC-TS achieves sublinear regret bound for linear contextual bandits.
New action poisoning attacks improve LinUCB's performance by changing action signals.
problem Improving understanding of adversarial attacks on contextual bandit algorithms.
method Proposed action poisoning attacks in white-box and black-box settings.
result Action poisoning attacks can force LinUCB to pull a target arm frequently with low cost.
We consider an online decision making setting known as contextual bandit problem, and propose an approach for improving contextual bandit performance by using an adaptive feature extraction (representation learning) based on online clustering. Our approach starts with an off-line pre-training on unlabeled history of co…
Contextual bandit algorithms are sensitive to the estimation method of the outcome model as well as the exploration method used, particularly in the presence of rich heterogeneity or complex outcome models, which can lead to difficult estimation problems along the path of learning. We study a consideration for the expl…
This paper achieves optimal regret bounds for locally private linear contextual bandit.
problem Designing locally private linear contextual bandit algorithms with optimal regret bounds.
method New algorithmic and analytical ideas, including mean absolute deviation analysis and layered principal component regression.
result Achieves an i l d e O ( T ) ilde O(\sqrt{T}) i l d e O ( T ) regret upper bound for locally private linear contextual bandit. A new algorithm for selecting top-k arms in extreme contextual bandits with improved efficiency.
problem Selecting top-k arms from a large set with contextual information and limited rewards.
method Proposes an algorithm for both non-extreme and extreme settings, using Inverse Gap Weighting and arm hierarchy models.
result Achieves improved regret guarantees for extreme settings with significant computational and statistical efficiency.
New algorithm for contextual bandits with linear constraints using regression.
problem Contextual bandits with packing and covering constraints.
method Modular Lagrangian approach via regression.
result First vanishing-regret guarantees for CBwLC (or CBwK).
New approach combines semi-supervised learning and bandits for better predictions.
problem Online semi-supervised learning with bandit feedback for applications like clinical trials and ad recommendations.
method Adjusted Graph Convolutional Network (GCN) for contextual bandits, with semi-supervised missing rewards imputation.
result Developed multi-GCN embedded contextual bandit algorithms verified on real-world datasets.
New algorithm for context bandits with continuous actions.
problem Efficient decision-making with unknown action structures.
method Reduction-style algorithm combining supervised learning.
result Proven to work in general and validated with experiments.
A new algorithm avoids worst-case outcomes in risky contexts.
problem Risk-averse behavior in contextual bandits is challenging.
method Developed a first risk-averse contextual bandit algorithm with online regret guarantees.
result First algorithm with an online regret guarantee for risk-averse contextual bandits.