Survey of MAB strategies for non-stationary reward distributions with delayed feedback.
problem Optimizing product availability in an online grocery pick-up platform with non-stationary and delayed reward feedback.
method Evaluation of ε ε ε -greedy, UCB1, Thompson Sampling, and a new adaptive technique (AG1) in MAB simulations. result AG1 outperforms traditional MAB strategies in minimizing regret for non-stationary and delayed feedback.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.
New TS algorithms improve performance in non-stationary multi-armed bandit problems.
problem Sequential decision-making with evolving action rewards.
method Sliding-window Thompson sampling approaches with different priors.
result Unified regret upper bound for arbitrary non-stationary MABs.
We extend Bayesian multi-armed bandit (MAB) algorithms beyond their original setting by making use of sequential Monte Carlo (SMC) methods. A MAB is a sequential decision making problem where the goal is to learn a policy that maximizes long term payoff, where only the reward of the executed action is observed. In the …
New algorithms detect changes in non-stationary MABs for better performance.
problem Non-stationary MAB environments where arm reward distributions change over time.
method Modular Detection Augmented Bandit (DAB) procedures with improved performance lower bounds.
result Modular DAB procedures achieve order-optimal regret bounds for various change detectors and bandit algorithms.
New algorithms reduce regret in non-stationary bandits with increasing payoffs.
problem Non-stationary bandits with monotonically increasing payoffs.
method R-ed-UCB for rested case and R-less-UCB for restless case.
result Regret bound of O ~ ( T 2 3 ) \widetilde{\mathcal{O}}(T^{\frac{2}{3}}) O ( T 3 2 ) under certain conditions. RAVEN-UCB addresses non-stationary MAB problems with tighter regret bounds.
problem Non-stationary environments in multi-armed bandits.
method Combines variance-aware adaptation with three innovations: confidence bounds, adaptive control, and recursive updates.
result Achieves tighter regret bounds than UCB1 and UCB-V.
New analysis improves Monte Carlo Tree Search for reinforcement learning.
problem Improving Monte Carlo Tree Search for reinforcement learning with finite simulations.
method Established polynomial concentration property of regret for non-stationary MABs, leading to a new UCB with polynomial bonus term.
result MCTS with polynomial bonus term requires nearly optimal sample size for learning value functions.
DS-TS adapts to abrupt and smooth changes in bandit problems.
problem Non-stationary multi-armed bandit problems with abrupt and smooth changes.
method Discounted Thompson Sampling with Gaussian priors.
result Achieves nearly optimal regret bound for both abrupt and smooth changes.
Paper designs a bandit algorithm without reward distribution info.
problem Designing bandit algorithms without reward distribution info.
method Alternates between greedy rule and forced exploration.
result Achieves substantial regret upper bounds.
A decentralized policy achieves logarithmic regret for multi-agent MAB problems with communication constraints.
problem Decentralized policy for multi-agent MAB problems with option availability and communication constraints.
method Upper Confidence Bound (UCB) algorithms with non-stationary stochastic communication protocol.
result Guaranteed logarithmic regret for non-fully connected spatial graphs with communication constraints.
Paper introduces novel Bandit algorithms for non-stationary environments in finance.
problem Non-stationary reward distributions in financial markets.
method Introduces Adaptive Discounted Thompson Sampling (ADTS) and Combinatorial Adaptive Discounted Thompson Sampling (CADTS) for non-stationary environments in portfolio optimization.
result Bandit Networks improve portfolio optimization performance by 20% compared to classical models.
The paper tackles non-stationary MAB with periodic rewards.
problem Non-stationary mean rewards over time in a business context.
method Combines Fourier analysis with confidence-bound learning to estimate periods and minimize regret.
result Proposes a near-optimal policy with a regret bound of O ( T ∑ k = 1 K T k ) O(\sqrt{T\sum_{k=1}^K T_k}) O ( T ∑ k = 1 K T k ) . New method tracks shifts in infinite-armed bandits without prior knowledge.
problem Tracking shifts in non-stationary infinite-armed bandits.
method Blackbox conversion of finite-armed MAB to infinite-armed non-stationary, randomized elimination.
result First parameter-free optimal regret bounds for all reservoir regularity regimes.
In a multi-armed bandit (MAB) problem a gambler needs to choose at each round of play one of K arms, each characterized by an unknown reward distribution. Reward realizations are only observed when an arm is selected, and the gambler's objective is to maximize his cumulative expected earnings over some given horizon of…
New algorithms control loss and constraints in uncertain, changing environments.
problem Adapting to adversarial constraints in uncertain, changing environments.
method Developed algorithms for constrained MAB problems with optimal rates of regret and positive constraint violation.
result Achieved optimal rates of regret and positive constraint violation under varying degrees of adversariality.
Proposes a new TS algorithm for non-stationary bandits using KS tests.
problem Non-stationary multi-armed bandit problems.
method Active detection of change points using KS tests and adaptive Thompson Sampling.
result Sub-linear regret demonstrated for the two-armed bandit case.
Study non-stationary bandits with resource constraints.
problem Maximize reward in a non-stationary environment with resource constraints.
method Propose new non-stationarity measure and use primal-dual analysis.
result Upper and lower bounds for non-stationary BwK problem.
We study the non-stationary stochastic multiarmed bandit (MAB) problem and propose two generic algorithms, namely, the limited memory deterministic sequencing of exploration and exploitation (LM-DSEE) and the Sliding-Window Upper Confidence Bound# (SW-UCB#). We rigorously analyze these algorithms in abruptly-changing a…
The paper tackles rested bandits with non-decreasing and concave rewards, deriving lower bounds and an efficient algorithm.
problem Studying the sample complexity and optimal strategies for rested bandits with specific reward properties.
method Deriving regret lower bounds and designing an efficient algorithm R-ed-UCB with theoretical and empirical analysis.
result An efficient algorithm R-ed-UCB with a regret bound of O ~ ( T 2 3 ) \widetilde{\mathcal{O}}(T^{\frac{2}{3}}) O ( T 3 2 ) under certain conditions. New method tracks significant shifts in nonparametric bandits.
problem Tracking significant changes in nonparametric contextual bandits.
method Proposed a notion of 'experienced significant shifts' to adapt to minimax rate without knowledge of change parameters.
result Experienced significant shifts count fewer changes than traditional metrics, leading to an adaptive algorithm.
Paper closes the gap in MP-MAB problems with novel adaptive communication and exploration.
problem Closing the gap between decentralized MP-MAB and natural centralized lower bound.
method BEACON: Batched Exploration with Adaptive COmmunicatioN, incorporating ADC and batched exploration.
result Proves logarithmic regret for a generalized MP-MAB problem.
We formulate the problem of sampling and recovering clustered graph signal as a multi-armed bandit (MAB) problem. This formulation lends naturally to learning sampling strategies using the well-known gradient MAB algorithm. In particular, the sampling strategy is represented as a probability distribution over the indiv…
A new bandit problem where experiments can be interrupted if results are not promising.
problem Interruptible multi-armed bandit problem with a threshold for cumulative reward.
method Formalized survival regret, identified key components (regret and probability of ruin), derived lower bounds and optimal policies.
result No policy can achieve sublinear survival regret, but optimal policies minimize survival regret in a Pareto sense.
In this work, we study recommendation systems modelled as contextual multi-armed bandit (MAB) problems. We propose a graph-based recommendation system that learns and exploits the geometry of the user space to create meaningful clusters in the user domain. This reduces the dimensionality of the recommendation problem w…
Trust-aware MAB improves learning performance by accounting for human deviation.
problem Learning performance suffers when humans deviate from recommended policies due to lack of trust.
method Integrates a dynamic trust model into MAB framework, establishing minimax regret and proposing a two-stage trust-aware procedure.
result Proves near-optimal statistical guarantees for trust-aware MAB algorithms.
New algorithm for multi-fidelity bandits reduces costs and improves regret.
problem Optimizing decisions with varying costs and accuracy in multi-fidelity bandits.
method Cost complexity bounds, algorithmic framework, elimination-based algorithm.
result New regret definition and matching upper and lower bounds for elimination-based algorithm.
New MAB algorithm for nonstationary, semiparametric reward models.
problem Nonstationary and non-linear reward models in sequential decision tasks.
method Contextual multi-armed bandit algorithm for semiparametric reward model.
result Proposed algorithm achieves a tight regret upper bound and performs well in simulations and real-world data.
New algorithm ensures consistent results in constrained MAB problems.
problem Achieving consistent results in constrained MAB problems.
method Developed replicable algorithms for constrained MAB problems using the optimism principle.
result Regret and constraint violation of replicable algorithms match those of non-replicable ones.
INF-clip optimizes heavy-tailed MAB problems with improved performance.
problem Optimizing multi-armed bandit problems with heavy-tailed rewards.
method INF-clip algorithm for adversarial and stochastic heavy-tailed MAB settings.
result INF-clip is optimal for linear and non-linear heavy-tailed stochastic MAB problems.
The Multi-Armed Bandits (MAB) framework highlights the tension between acquiring new knowledge (Exploration) and leveraging available knowledge (Exploitation). In the classical MAB problem, a decision maker must choose an arm at each time step, upon which she receives a reward. The decision maker's objective is to maxi…
We extend the classic multi-armed bandit (MAB) model to the setting of noncompliance, where the arm pull is a mere instrument and the treatment applied may differ from it, which gives rise to the instrument-armed bandit (IAB) problem. The IAB setting is relevant whenever the experimental units are human since free will…
New MAB model for online caching costs.
problem Learning costs of cached items online.
method Synchronization bandits, MirrorSync algorithm.
result Adversarial regret of O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) for MirrorSync. Chronological Causal Bandits (CCB) tackles dynamic causal decision-making.
problem Dynamic causal decision-making in a system where rewards depend on past interventions.
method Introduces a new MAB problem (Chronological Causal Bandit) where rewards are influenced by a dynamic causal model.
result Early findings show the CCB can transfer information between sequential MABs.
New model for multi-armed bandits with growing arms.
problem Balancing exploration and exploitation in a growing set of arms.
method Introduces Ballooning Multi-Armed Bandits (BL-MAB) and analyzes existing algorithms.
result Achieves sub-linear regret under certain conditions.
Develops Thompson Sampling algorithms for mean-variance bandits.
problem Risk in online decision making systems.
method Thompson Sampling algorithms for mean-variance MAB with comprehensive regret analyses.
result Achieves best known regret bounds for mean-variance MABs and information-theoretic bounds in some regimes.
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options against control pills for different diseases, or when internet companies test the…
Improved Boltzmann exploration reduces regret in stochastic MAB problems.
problem Improving regret in stochastic multi-armed bandit problems.
method A modified Boltzmann exploration with a doubling trick.
result Achieves O ( K log 1 + α T ) O(K\log^{1+α} T) O ( K log 1 + α T ) regret, improving on previous results. Solves Multivariate-MAB problem with path planning and Thompson sampling.
problem Exponential exploding issue in multivariate Multi-Armed Bandit (Multivariate-MAB) problem.
method Path planning framework using decision graphs and Thompson sampling for heuristic arm selection.
result Achieves faster convergence speed, better efficient arm allocation, and lower cumulative regret.
This paper deals with bandit online learning problems involving feedback of unknown delay that can emerge in multi-armed bandit (MAB) and bandit convex optimization (BCO) settings. MAB and BCO require only values of the objective function involved that become available through feedback, and are used to estimate the gra…
Incentivized exploration improves MAB performance under reward drift.
problem Improving exploration in multi-armed bandits with biased feedback.
method Analysis of three MAB algorithms (UCB, ε-Greedy, Thompson Sampling) under drifted reward feedback.
result All algorithms achieve O ( log T ) \mathcal{O}(\log T) O ( log T ) regret and compensation under drifted reward. New MAB model incentivizes user arm-pulling with self-reinforcing preferences.
problem Balancing exploration and exploitation in recommender systems with incentivized user preferences.
method Proposes a new MAB model with random arm selection and two policies: At-Least- n n n Explore-Then-Commit and UCB-List. result Achieves O ( l o g T ) O(log T) O ( l o g T ) expected regret and O ( l o g T ) O(log T) O ( l o g T ) expected payment over a time horizon T T T . New L2 regularization improves softmax MAB performance.
problem Improving softmax MAB performance with vanishing regularization.
method L2 regularization with vanishing parameter analyzed and proven convergent.
result Vanishing L2 regularization makes softmax MAB more numerically advantageous.
A scheme robust to action erasures improves MAB performance.
problem Learning in multi-arm bandits with erasure channels.
method Proposes a scheme to work on top of any MAB algorithm.
result Worst-case regret is at most a factor of O ( 1 / 1 − ε ) O(1/\sqrt{1-ε}) O ( 1/ 1 − ε ) away from no-erasure case. This paper analyzes the multi-armed bandit model using path-integral methods.
problem Understanding the stochastic dynamics and optimal strategies in multi-armed bandit problems.
method Path-integral analysis of statistical physics.
result Emergence of multimodal regret distribution with large regrets from exploitation of sub-optimal arms.
Paper proposes CVaR-TS for risk-constrained MAB problems.
problem Risk in decision-making complicates reward maximization in MAB problems.
method Risk measure CVaR is used, and Thompson Sampling is adapted for CVaR.
result CVaR-TS outperforms other L/UCB-based algorithms in risk-constrained MAB settings.
We study a policy gradient method with L2 regularization for MAB problems.
problem Improving policy gradient methods for MAB problems with regularization.
method Investigate convergence of a policy gradient algorithm with L2 regularization for MAB.
result Prove convergence under appropriate technical hypotheses and show practical improvements.
The paper studies MAB problems with LDP to balance privacy and service quality.
problem Balancing privacy and service quality in multi-armed bandit systems.
method Investigates regret minimization for MAB with LDP guarantee, proving lower bounds and proposing matching upper bounds algorithms.
result Regret upper bounds match lower bounds up to constant factors for MAB algorithms with LDP guarantee.