Thompson sampling used for linear bandits with normal-gamma priors.
problem Optimizing decisions in uncertain environments with linear dependencies and unknown parameters.
method Bayesian Thompson sampling with multivariate normal-gamma priors.
result Derivation of a Bayesian regret bound for the approach.
Paper improves Thompson Sampling for linear contextual bandits.
problem Empirical Thompson Sampling does not achieve optimal regret bounds.
method Develops a novel estimator with adaptive data augmentation and coupling.
result Achieves nearly minimax optimal performance.
New Thompson sampling algorithm for stochastic partial monitoring achieves logarithmic regret.
problem Limited feedback in sequential learning problems.
method Developed a novel Thompson-sampling-based algorithm to sample from the posterior distribution exactly.
result Achieved logarithmic regret bound of O(log T) for a linearized variant of the problem.
Improved Thompson Sampling for high-dimensional sparse bandits.
problem Stochastic linear contextual bandits with high-dimensional features.
method Thompson Sampling with spike-and-slab priors and variational inference.
result Nearly optimal upper bound on expected cumulative regret.
Randomized exploration in linear bandits achieves optimal regret bounds.
problem Optimizing exploration in high-dimensional linear bandit problems.
method Analysis of Thompson sampling without forced optimism.
result Randomized exploration algorithms achieve an O ( d n log ( n ) ) O(d\sqrt{n} \log(n)) O ( d n log ( n )) regret bound in smooth, strongly convex action spaces. MOL-TS uses Thompson Sampling for multi-objective linear bandits with Pareto guarantees.
problem Optimizing multiple conflicting objectives in linear contextual bandits.
method Proposes MOL-TS, a Thompson Sampling algorithm with Pareto regret guarantees.
result Achieves a worst-case Pareto regret bound of O ~ ( d 3 / 2 T ) \widetilde{O}(d^{3/2}\sqrt{T}) O ( d 3/2 T ) . Modified Meta-TS for linear contextual bandits reduces regret.
problem Optimizing decision-making in dynamic environments with context vectors.
method Meta-TSLB algorithm for linear contextual bandits, analyzing Bayes regret.
result Derives an O ( ( m + log ( m ) ) n log ( n ) ) O((m+\log(m))\sqrt{n\log(n)}) O (( m + log ( m )) n log ( n ) ) bound on Bayes regret. In this note, we present a version of the Thompson sampling algorithm for the problem of online linear generalization with full information (i.e., the experts setting), studied by Kalai and Vempala, 2005. The algorithm uses a Gaussian prior and time-varying Gaussian likelihoods, and we show that it essentially reduces …
VITS improves Thompson Sampling for contextual bandits with efficient approximate inference.
problem Intractable posterior sampling in traditional Thompson Sampling for contextual bandits.
method VITS uses Gaussian Variational Inference for efficient posterior approximations.
result VITS achieves sub-linear regret bound similar to traditional TS.
Thompson Sampling fails to perform well in high dimensions.
problem Thompson Sampling's suboptimality in high-dimensional combinatorial semi-bandits.
method Analysis of TS for combinatorial semi-bandits, including non-linear and linear reward functions, with Bernoulli rewards and uniform priors.
result TS's regret scales exponentially in the ambient dimension and minimax regret scales almost linearly in high dimensions.
Thompson Sampling shows polynomial regret for combinatorial semi-bandits with subgaussian rewards.
problem Finding optimal solutions in combinatorial semi-bandits with suboptimal sampling.
method Proposes Thompson Sampling with polynomial regret for linear combinatorial semi-bandits.
result Demonstrates 'mismatched sampling paradox' where knowing distributions can lead to worse performance.
Ensemble sampling approximates Thompson sampling for complex models.
problem Computational intractability of exact posterior distributions.
method Thompson sampling approximation with information-theoretic concepts.
result Established a first rigorous regret bound for ensemble sampling.
FGTSVA improves Thompson Sampling for contextual bandits with optimal variance-aware regret.
problem Optimizing regret bounds for Thompson Sampling in contextual bandits.
method Developed FGTSVA, a variance-aware Thompson Sampling algorithm for contextual bandits with a new decoupling coefficient.
result Achieved optimal regret bound of i l d e O ( d c ⋅ log ∣ F ∣ ∑ t = 1 T σ t 2 + d c ) ilde{O}(\sqrt{\mathrm{dc}\cdot\log|\mathcal{F}|\sum_{t=1}^Tσ_t^2}+\mathrm{dc}) i l d e O ( dc ⋅ log ∣ F ∣ ∑ t = 1 T σ t 2 + dc ) . Improved Thompson Sampling reduces regret in contextual bandits and reinforcement learning.
problem Thompson Sampling's exploration is insufficient in some contexts.
method Developed Feel-Good Thompson Sampling to address exploration issues.
result Feel-Good Thompson Sampling reduces regret compared to standard Thompson Sampling.
Ensemble++ uses shared-factor ensembles to scale Thompson Sampling for linear and nonlinear bandits.
problem Computational challenges in Thompson Sampling for large-scale or non-conjugate settings.
method Ensemble++ with shared-factor architecture and random linear combinations.
result Ensemble++ achieves comparable regret to exact Thompson Sampling with significantly smaller ensemble sizes.
Thompson Sampling bounds for contextual bandits with sub-Gaussian rewards.
problem Improving the performance of Thompson Sampling in contextual bandits with sub-Gaussian rewards.
method Proved comprehensive bounds on Thompson Sampling expected cumulative regret based on mutual information and lifted information ratio for sub-Gaussian rewards.
result Explicit regret bounds for various contextual bandit scenarios.
New algorithm learns LQR with O ( T ) O(\sqrt{T}) O ( T ) regret using Langevin dynamics and excitation.
problem Learning LQR with a O ( T ) O(\sqrt{T}) O ( T ) regret bound. method Thompson sampling with Langevin dynamics and excitation mechanism.
result Achieved O ( T ) O(\sqrt{T}) O ( T ) regret bound for LQR learning. The paper tightens the regret rate for linear bandit problems.
problem Bayesian regret in linear bandit problems.
method Information-theoretic framework and chaining argument.
result Established a new bound with a tight rate of O ( d T ) O(d\sqrt{T}) O ( d T ) . Extends Thompson sampling for RL with fewer episodes.
problem Limited episodes in RL settings.
method Batch Bayesian optimization over episodes to learn action bias terms.
result Significantly outperforms standard Thompson sampling.
TS-UCB improves Thompson Sampling with minimal extra computation.
problem Online decision problems with bandit feedback.
method TS-UCB uses posterior samples and upper confidence bounds to select arms.
result TS-UCB achieves lower regret on various datasets.
Hypermodels improve exploration efficiency and accuracy.
problem Efficiently approximating Thompson sampling with large ensembles.
method Introducing hypermodels as a generalization of ensembles, including linear and neural network hypermodels.
result Hypermodels enable more accurate exploration and performance gains over Thompson sampling.
The paper addresses frequentist regret of Linear Thompson Sampling in stochastic linear bandits.
problem The frequentist regret of Linear Thompson Sampling (LinTS) is worse than its Bayesian counterpart.
method The paper proves the fundamental nature of the frequentist regret bound for LinTS and proposes a data-driven version of LinTS to achieve minimax optimal frequentist regret.
result The frequentist regret bound for LinTS is O ~ ( d d T ) \widetilde{\mathcal{O}}(d\sqrt{dT}) O ( d d T ) , which is the best possible under certain conditions. ES reduces high-probability regret in stochastic linear bandits.
problem High-probability regret in stochastic linear bandits.
method Linear ensemble sampling with standard Gaussian perturbations, analyzing m = Θ ( d log n ) m=Θ(d\log n) m = Θ ( d log n ) ensemble size. result ES achieves i l d e O ( d 3 / 2 n ) ilde O(d^{3/2}\sqrt n) i l d e O ( d 3/2 n ) high-probability regret, closing the gap to Thompson sampling. Batch Thompson Sampling reduces exploration-exploitation trade-off in online decision making.
problem Balancing exploration and exploitation in online decision making.
method Introducing a batch Thompson Sampling framework for stochastic multi-arm bandit and linear contextual bandit problems.
result Achieves asymptotic regret bound with O ( log T ) O(\log T) O ( log T ) batch queries, significantly reducing interactions. The paper extends Thompson Sampling to infinite action spaces using information theory.
problem Addressing the limitation of finite action spaces in Thompson Sampling.
method Information-theoretic analysis, extending rate-distortion theory to infinite action spaces.
result Derives a near-optimal regret bound for bandits with infinite and continuous action spaces.
LMC-TS uses MCMC for efficient posterior sampling in contextual bandits.
problem Efficiency of Thompson sampling for high-dimensional contextual bandits.
method Langevin Monte Carlo for direct posterior sampling.
result LMC-TS achieves sublinear regret bound for linear contextual bandits.
Thompson Sampling is one of the oldest heuristics for multi-armed bandit problems. It is a randomized algorithm based on Bayesian ideas, and has recently generated significant interest after several studies demonstrated it to have better empirical performance compared to the state-of-the-art methods. However, many ques…
MixTS uses a mixture prior to analyze Thompson Sampling in multi-task learning.
problem Analyzing Thompson Sampling in environments with uncertain and multi-class problems.
method Developed MixTS by incorporating a mixture prior into Thompson Sampling and using a novel proof technique for mixture distributions.
result Proved Bayes regret bounds for MixTS in linear bandits and finite-horizon reinforcement learning.
We study the effects of approximate inference on the performance of Thompson sampling in the k k k -armed bandit problems. Thompson sampling is a successful algorithm for online decision-making but requires posterior inference, which often must be approximated in practice. We show that even small constant inference error …
Improved Bayesian regret bound for linear Thompson sampling with general distributions.
problem Proving an improved Bayesian regret bound for linear Thompson sampling with general distributions.
method Generalized elliptical potential lemma for non-Gaussian noise and prior distributions.
result Minimax optimal regret bound for changing action sets with general prior and noise distributions.
Bayesian methods improve drug discovery experiment design.
problem Optimizing drug screening experiments in high-dimensional data.
method Bayesian inference and optimisation with upper confidence bound algorithms, Thompson sampling, and sparse tree search.
result Sparse tree search techniques outperform other methods in drug toxicity screening.
PFN-TS uses Thompson sampling with PFNs to improve contextual bandit performance.
problem Improving contextual bandit performance using Thompson sampling with prior-data fitted networks.
method PFN-TS converts PFN posterior predictives into mean-reward samples using a subsampled predictive central limit theorem.
result PFN-TS achieves the best average rank across nonlinear synthetic and OpenML classification-to-bandit benchmarks.
New algorithm explores optimally using same primitives as Thompson Sampling.
problem Optimal exploration of unknown parameter vectors with noisy measurements.
method Algorithm leveraging sampling and argmax oracles.
result Achieves exponential convergence rate, optimal among all allocations.
Paper improves Bayesian regret bounds for Thompson Sampling in reinforcement learning.
problem Improving Bayesian regret bounds for Thompson Sampling in reinforcement learning.
method Using a discrete set of surrogate environments and posterior consistency analysis, the authors derive an upper bound of order O ( H d l 1 T ) O(H\sqrt{d_{l_1}T}) O ( H d l 1 T ) . result The derived upper bound of O ( H d l 1 T ) O(H\sqrt{d_{l_1}T}) O ( H d l 1 T ) is a significant improvement over previous bounds. We consider the exploration-exploitation tradeoff in linear quadratic (LQ) control problems, where the state dynamics is linear and the cost function is quadratic in states and controls. We analyze the regret of Thompson sampling (TS) (a.k.a. posterior-sampling for reinforcement learning) in the frequentist setting, i.…
Thompson sampling provides a solution to bandit problems in which new observations are allocated to arms with the posterior probability that an arm is optimal. While sometimes easy to implement and asymptotically optimal, Thompson sampling can be computationally demanding in large scale bandit problems, and its perform…
Graph neural Thompson Sampling improves online decision-making for graph data.
problem Online decision-making with graph-structured rewards.
method GNN-TS algorithm using GNN for mean reward estimation and graph neural tangent features for uncertainty.
result GNN-TS achieves a state-of-the-art regret bound of i l d e O ( ( i l d e d T ) 1 / 2 ) ilde{\mathcal{O}}(( ilde{d} T)^{1/2}) i l d e O (( i l d e d T ) 1/2 ) . Thompson Sampling tackles noisy context in stochastic bandits.
problem Designing an action policy for noisy, corrupted contexts in stochastic bandits.
method Introducing a Thompson Sampling algorithm for Gaussian bandits with Gaussian context noise, adopting an information-theoretic analysis.
result Demonstrates the Bayesian regret of the proposed algorithm concerning the oracle's action policy.
Optimizes Thompson sampling policies using policy gradient methods.
problem Improving Thompson sampling in bandit problems.
method Applies policy gradient algorithms to optimize Thompson sampling policies.
result Direct policy search on Thompson sampling improves performance.
A new algorithm combines SGD and Thompson Sampling for contextual bandits.
problem Finding efficient algorithms for contextual bandits with low time and memory complexity.
method Online Stochastic Gradient Descent (SGD) combined with Thompson Sampling.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret with linear time complexity in T T T and d d d . We derive an alternative proof for the regret of Thompson sampling (\ts) in the stochastic linear bandit setting. While we obtain a regret bound of order O ~ ( d 3 / 2 T ) \widetilde{O}(d^{3/2}\sqrt{T}) O ( d 3/2 T ) as in previous results, the proof sheds new light on the functioning of the \ts. We leverage on the structure of the problem to show …
LinConTS improves regret and constraint violations in probabilistic linearly constrained bandits.
problem Maximizing cumulative reward under probabilistic linear constraints.
method LinConTS, a Thompson Sampling-based algorithm for bandits with linear constraints.
result LinConTS achieves O(log T) regret and constraint violations for suboptimal arms.
A new bandit strategy improves learning speed at the cost of accuracy.
problem Learning speed vs. accuracy trade-off in multi-armed bandits.
method Weighted Thompson Sampling (WTS) for multi-armed bandits with Gaussian feedback.
result Upper bound matches log(T) lower bound, solving fundamental problem.
Stochastic linear bandits are a natural and simple generalisation of finite-armed bandits with numerous practical applications. Current approaches focus on generalising existing techniques for finite-armed bandits, notably the optimism principle and Thompson sampling. While prior work has mostly been in the worst-case …
Optimistic actor-critic tackles linear MDPs with parametric policies.
problem Theoretical limitations of existing actor-critic methods for linear MDPs.
method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.
result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.
BFTS uses Bayesian Additive Regression Trees for improved personalized mobile health interventions.
problem Adapting to complex, non-linear user behaviors in personalized mobile health interventions.
method Bayesian Forest Thompson Sampling (BFTS) integrates Bayesian Additive Regression Trees (BART) into the exploration loop of contextual bandits.
result BFTS achieves state-of-the-art regret on tabular benchmarks and improves engagement rates by over 30% in a behavioral intervention study.
Study ranking in generalized linear bandits with position and item dependencies.
problem Complex reward function due to position and item dependencies in recommendation systems.
method Model position and item dependencies, design UCB and Thompson Sampling algorithms.
result Generalizes existing studies in position dependencies and graph theory.
The design and performance analysis of bandit algorithms in the presence of stage-wise safety or reliability constraints has recently garnered significant interest. In this work, we consider the linear stochastic bandit problem under additional \textit{linear safety constraints} that need to be satisfied at each round.…