New algorithms tackle latent bandit problems with lower regret.
problem Learning optimal actions in a latent state setting.
method General algorithms based on UCBs and Thompson sampling, aware of model uncertainty.
result Our methods have lower regret than classic bandit policies when latent states are fewer than actions.
New algorithms use offline data to improve online decision-making with latent states.
problem Accelerating online sequential decision-making with latent states in offline data.
method Design end-to-end latent bandit algorithms for linear latent contextual bandits, learning latent subspace offline and using it online.
result Proves minimax optimal regret guarantees for online algorithms and practical efficiency.
Proposes a method to learn from historical data for personalized decision-making.
problem Sample hunger in sequential decision-making algorithms for personalized medicine.
method Identifiable latent bandit framework using nonlinear independent component analysis.
result Optimal decision-making with shorter exploration time than classical bandits.
Optimal algorithm for latent bandits with cluster structure reduces regret to nearly optimal.
problem Maximizing cumulative rewards in a multi-armed bandit problem with latent clusters.
method LATTICE algorithm exploiting cluster structure and arm information.
result Minimax optimal regret of O ( ( M + N ) T ) O(\sqrt{(\mathsf{M}+\mathsf{N})\mathsf{T}}) O ( ( M + N ) T ) with O ( log T ) O(\log{\mathsf{T}}) O ( log T ) calls to matrix completion oracle. Transfer learning for bandits with latent Lipschitz continuity.
problem Learning to transfer structural information from prior tasks to new tasks.
method Proposes a framework to estimate Lipschitz constant from prior tasks and apply it to new tasks.
result Regret bound close to oracle algorithm with full knowledge of Lipschitz constant under mild assumptions.
A new framework for structured bandits using influence diagrams and variational Thompson sampling.
problem Complex statistical dependencies in structured bandit problems.
method Influence diagram framework, variational Thompson sampling, tracking structured posterior distribution.
result Empirically evaluated algorithms perform as well as or better than existing baselines.
Unified approach to correlated multi-armed bandits reduces regret significantly.
problem Correlated rewards in multi-armed bandits.
method Developed a unified approach to leverage reward correlations and presented algorithms with rigorous analysis.
result C-UCB algorithm pulls non-competitive arms only O(1) times, improving over classic algorithms.
New framework tackles stochastic latent subgroup heterogeneity in online decision-making.
problem Stochastic latent heterogeneity in online decision-making where individual responses vary with unobserved subgroups.
method Latent heterogeneous bandit framework using EM-greedy algorithm to learn subgroup probabilities and reward parameters.
result Achieves optimal estimation and classification guarantees, revealing a fundamental stochastic barrier in online decision-making.
Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
New algorithm learns causal graph to minimize regret in bandits without full structure.
problem Learning optimal decisions in bandits with unknown causal graph and latent confounders.
method Two-stage approach: first learns ancestors and necessary confounders, second applies standard bandit algorithm.
result No full causal structure needed for optimal decisions; only necessary confounders are crucial.
Algorithm maximizes user rewards under per-item budget constraints.
problem Maximizing cumulative rewards in collaborative bandits with budget constraints.
method Collaborative algorithm B-LATTICE that clusters users and collaborates across groups.
result Achieves sub-linear regret bounds matching minimax bounds.
Motivated by online recommendation and advertising systems, we consider a causal model for stochastic contextual bandits with a latent low-dimensional confounder. In our model, there are L L L observed contexts and K K K arms of the bandit. The observed context influences the reward obtained through a latent confounder var…
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) regret. We stabilize the Kumaraswamy distribution for efficient sampling and differentiation.
problem Numerical instabilities in the Kumaraswamy distribution's inverse CDF and log-pdf.
method Identified and resolved numerical issues, introduced a stabilized KS distribution.
result Stabilized Kumaraswamy distribution supports efficient sampling and differentiation.
Decentralized learning for matching markets with time-varying preferences.
problem Matching between competing agents and supply arms with time-varying preferences.
method Linear contextual bandit framework, learning algorithms to identify latent environment and stable matchings.
result Achieve instance-dependent logarithmic regret, applicable for large markets.
We consider dynamic pricing with many products under an evolving but low-dimensional demand model. Assuming the temporal variation in cross-elasticities exhibits low-rank structure based on fixed (latent) features of the products, we show that the revenue maximization problem reduces to an online bandit convex optimiza…
We consider a novel multi-armed bandit framework where the rewards obtained by pulling the arms are functions of a common latent random variable. The correlation between arms due to the common random source can be used to design a generalized upper-confidence-bound (UCB) algorithm that identifies certain arms as $non-c…
A new estimator reduces variance in slate bandit OPE.
problem Large action spaces in slate bandits cause high variance in OPE.
method Develops Latent IPS (LIPS) to optimize slate abstractions for low variance and bias.
result LIPS substantially outperforms existing estimators in scenarios with non-linear rewards and large slate spaces.
New algorithm for partially observable contexts in finance.
problem Decision making based on partially observable, correlated market information.
method EMKF-Bandit algorithm integrating system identification, filtering, and bandit algorithms.
result Sub-linear regret under conditions on filtering.
New definition resolves ambiguity in non-stationary bandit classification.
problem Ambiguity in classifying non-stationary bandits using existing definitions.
method Introducing a formal definition that resolves ambiguity and provides a unified approach.
result Unified approach applicable to both Bayesian and frequentist formulations, resolves classification issues.
Jointly tackles assortment and pricing in retail, using bandit models.
problem Maximizing revenue or profit in retail through optimal assortment and pricing.
method Contextual bandits with a flexible, interpretable model for high-dimensional contexts and actions.
result Proves lower regret compared to state-of-the-art methods in various bandit and pricing models.
We introduce data-driven decision-making algorithms that achieve state-of-the-art \emph{dynamic regret} bounds for non-stationary bandit settings. These settings capture applications such as advertisement allocation, dynamic pricing, and traffic network routing in changing environments. We show how the difficulty posed…
Thompson Sampling is at most twice as bad as any other policy in Bayesian bandit models.
problem Optimizing selection of the best arm in Bayesian bandit models with independent latent processes.
method Thompson Sampling approach applied to models with independent latent arm processes.
result Thompson Sampling makes at most twice the expected number of mistakes compared to any other policy.
Thompson sampling, a Bayesian method for balancing exploration and exploitation in bandit problems, has theoretical guarantees and exhibits strong empirical performance in many domains. Traditional Thompson sampling, however, assumes perfect compliance, where an agent's chosen action is treated as the implemented actio…
New method tackles confounded bandit problems with dual instrumental variables.
problem Confounded contextual bandit problems where noise affects both contexts and rewards.
method Dual instrumental variable regression applied to reproducing kernel Hilbert spaces.
result Near-optimal convergence rate and computationally efficient algorithms proved.
Algorithm finds latent structure in value functions for improved reinforcement learning.
problem Finding latent structure in value functions for efficient reinforcement learning.
method Proposes a practical algorithm using two posterior distributions over state abstractions and abstract-state values.
result Substantial performance gains in multi-task settings where tasks share a common, low-dimensional representation.
Evaluating novel contextual bandit policies using logged data is crucial in applications where exploration is costly, such as medicine. But it usually relies on the assumption of no unobserved confounders, which is bound to fail in practice. We study the question of policy evaluation when we instead have proxies for th…
Algorithm selects k arms from context-dependent options using Plackett-Luce model.
problem Selecting k arms from context-dependent options with Plackett-Luce feedback.
method Proposes CPPL algorithm inspired by UCB, evaluated on synthetic and real data.
result Demonstrates effectiveness of CPPL algorithm in online algorithm selection.
BLOB combines organic and bandit signals for better user interest estimation.
problem Combining organic and bandit signals for improved user interest estimation.
method Bayesian Latent Organic Bandit (BLOB) model using variational auto-encoders and local re-parametrization.
result BLOB outperforms organic and bandit-based methods in both organic and bandit-rich environments.
New algorithm identifies best intervention without graph knowledge.
problem Finding best intervention in causal bandit setting with unknown graph.
method Additive combinatorial linear bandit problem with action-elimination algorithm.
result One can identify best intervention without explicitly learning graph parents.
A new algorithm for bandits with hierarchical rewards.
problem Learning from correlated rewards in complex hierarchies.
method Hierarchical Thompson Sampling (HierTS) for Gaussian hierarchies.
result Hierarchical Thompson Sampling reduces regret by non-constant factors in the number of actions.
We consider recommendation systems that need to operate under wireless bandwidth constraints, measured as number of broadcast transmissions, and demonstrate a (tight for some instances) tradeoff between regret and bandwidth for two scenarios: the case of multi-armed bandit with context, and the case where there is a la…
Algorithm minimizes regret in dueling bandits with contextualized utilities.
problem Minimizing regret in dueling bandits with context-dependent utilities.
method Proposes CoLSTIM algorithm based on perturbed utility estimates.
result Achieves regret of order i l d e O ( d T ) ilde O(\sqrt{dT}) i l d e O ( d T ) . New algorithm catches moving subspaces in bandit problems.
problem Adapt to changing low-dimensional latent subspaces in bandit settings.
method Piecewise-stationary low-rank linear contextual bandits with CUSUM-style boundary detection.
result Achieves intrinsic rank dynamic regret rate of O ( r T ) O(r\sqrt{T}) O ( r T ) . PFN-TS uses Thompson sampling with PFNs to improve contextual bandit performance.
problem Improving contextual bandit performance using Thompson sampling with prior-data fitted networks.
method PFN-TS converts PFN posterior predictives into mean-reward samples using a subsampled predictive central limit theorem.
result PFN-TS achieves the best average rank across nonlinear synthetic and OpenML classification-to-bandit benchmarks.
New algorithm tackles dynamic query routing to multiple embedding models.
problem Dynamic query routing to multiple embedding models under adversarial conditions.
method Formalized as adversarial contextual linear bandit with low-rank experts, proposed HPG algorithm.
result HPG algorithm achieves linearized policy regret of i l d e O ( s M T ) ilde{\mathcal O}(s\sqrt{M T}) i l d e O ( s M T ) . We study multi-armed bandit problems with graph feedback, in which the decision maker is allowed to observe the neighboring actions of the chosen action, in a setting where the graph may vary over time and is never fully revealed to the decision maker. We show that when the feedback graphs are undirected, the original …
New Thompson sampling uses local uncertainty for better decision making.
problem Sequential decision making with exploration-exploitation dilemma.
method Proposes a new probabilistic modeling framework using local latent variable uncertainty for Thompson sampling, with variational inference and semi-implicit structure.
result Thompson sampling guided by local uncertainty achieves state-of-the-art performance with low computational complexity.
New method learns decisions from collective preferences without individual covariates.
problem Making decisions online without individual covariates.
method Collaborative filtering, matrix completion bandit, ε-greedy policy, online gradient descent, inverse propensity weighting.
result Method outperforms benchmarks and reveals new discoveries.
Adaptive algorithms minimize regret in matching markets with contextual arm preferences.
problem Minimizing regret in matching markets with context-dependent player utilities.
method Developed adaptive algorithms for stochastic and adversarial contexts, providing upper and lower bounds.
result Achieved sublinear regret bounds for both stochastic and adversarial contexts.
We address the cold start problem in recommendation systems assuming no contextual information is available neither about users, nor items. We consider the case in which we only have access to a set of ratings of items by users. Most of the existing works consider a batch setting, and use cross-validation to tune param…
New method learns near-optimal policies with polynomial samples in A and H.
problem Episodic latent MABs with partial observations are challenging.
method Experiment design and method-of-moments approach.
result Polynomial samples in A and H for near-optimal policy learning.
REN addresses uncertainty in user feedbacks for better recommendation systems.
problem Recurrent neural networks focus solely on item relevance, neglecting diverse item exploration.
method Proposes REN, a new type of recurrent neural network that balances relevance and exploration while accounting for representation uncertainty.
result REN achieves satisfactory long-term rewards on synthetic and real-world recommendation datasets, outperforming state-of-the-art models.
TOFU-POV tackles partially observed linear bandits, achieving sublinear regret with low-dimensional action vectors.
problem Stochastic linear bandits with partially observed actions in settings like recommendation and healthcare.
method TOFU-POV estimates latent action subspace, imputes missing actions, and runs OFUL in low-dimensional coordinates.
result TOFU-POV achieves T \sqrt{T} T regret scaling with intrinsic subspace dimension, improving upon natural baselines. We study the problem of online influence maximization in social networks. In this problem, a learner aims to identify the set of "best influencers" in a network by interacting with it, i.e., repeatedly selecting seed nodes and observing activation feedback in the network. We capitalize on an important property of the i…
The paper tackles combinatorial pure exploration with various feedback structures and proposes efficient algorithms.
problem Identifying the optimal action in a combinatorial space with limited feedback and nonlinear rewards.
method Designs polynomial-time adaptive algorithms for CPE-BL and CPE-PL, providing sample complexity analyses.
result The proposed algorithms achieve sample complexity close to lower bounds and outperform existing methods.
New method optimizes policies in non-stationary environments.
problem Optimizing policies in non-stationary, context-dependent environments.
method Two-phase approach: offline learning and online adaptation.
result Our method outperforms existing approaches in both synthetic and real-world datasets.
Many problems in computer vision and recommender systems involve low-rank matrices. In this work, we study the problem of finding the maximum entry of a stochastic low-rank matrix from sequential observations. At each step, a learning agent chooses pairs of row and column arms, and receives the noisy product of their l…