Study on indexability of restless multi-armed bandits and rollout policy performance.
problem Maximizing discounted rewards in finite state restless multi-armed bandit problems.
method Decouple the problem into single-armed restless bandits, analyze using value iteration, and compare with Whittle index policy.
result Demonstrates conditions for indexability and compares performance of index policy and rollout policy.
NeurWIN learns Whittle indices for restless bandits using deep reinforcement learning.
problem Finding optimal solutions for restless bandits with complex transition kernels.
method NeurWIN uses a neural network to learn Whittle indices, leveraging the Whittle indices' mathematical properties.
result NeurWIN outperforms other RL algorithms in solving three recently studied restless bandit problems.
Graph-Triggered Bandits unify rested and restless bandits with graph-defined arm interactions.
problem Modeling sequential decision-making problems with evolving arm rewards.
method Graph-Triggered Bandits (GTBs) framework that generalizes rested and restless bandits using a graph.
result Rested and restless bandits are special cases of GTBs for suitable graphs.
New algorithm uses Whittle index to improve Q-learning for restless bandits.
problem Optimizing decision-making in multiarmed restless bandits with average reward.
method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.
result Demonstrates significant computational gains and excellent empirical performance.
This paper analyzes Thompson Sampling in restless bandits with unknown parameters.
problem Analyzing performance of Thompson Sampling in restless bandits with unknown parameters.
method Proved a Bayesian regret bound of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) for Thompson Sampling in episodic restless bandits with unknown parameters. result Proved a regret bound of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) for Thompson Sampling in restless bandits with unknown parameters. New algorithm handles both decaying and non-decaying bandit problems.
problem Decaying rewards in bandit problems.
method RAW-UCB algorithm for both rested and restless rotting bandits.
result Achieves near-optimal regret in both rested and restless rotting bandits.
The paper tackles restless bandits with limited observation, proposing a method to analyze and approximate their optimal strategies.
problem Restless bandits with limited observation.
method General probabilistic model, PCL analysis, and approximation process.
result The proposed method can transform the problem into a finite-state problem, enabling the use of existing algorithms.
Thompson Sampling reduces regret in non-episodic restless bandits.
problem Learning optimal policies in time-varying reward systems.
method Variant of Thompson Sampling applied to infinite horizon restless bandits.
result Sub-linear regret bound of O ( T log T ) O(\sqrt{T}\log T) O ( T log T ) proved for non-episodic case. Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
A new algorithm for restless bandits handles long-range dependencies.
problem Generalization of linear bandits with time-dependent parameters.
method LinMix-UCB algorithm with Berbee's coupling lemma.
result Sub-linear regret of $\mathcal{O}\left(\sqrt{d n\mathrm{polylog}(n) }
ight)$ .
Study best arm identification in restless bandits with unknown TPMs.
problem Identify the best arm with fixed confidence in restless bandits with unknown TPMs.
method Proposed a policy for best arm identification and proved its expected stopping time matches the lower bound.
result The state-action visitation proportions match the optimal proportions under any asymptotically optimal policy.
New model predicts optimal control for restless bandit problems.
problem Optimizing control in restless bandit problems with minimal assumptions.
method Model Predictive Control with rolling horizon linear programming.
result Sub-optimality gap of O(1/√N) under general conditions, and exp(-Ω(N)) under local-stability condition.
The paper develops a UCB-based method for a specific class of restless bandit problems.
problem Real-world restless bandit problems are complex and hard to solve optimally.
method Modified UCB algorithm for φ \varphi φ -mixing stationary pay-off distributions. result UCB-based method provides good approximate solutions under certain conditions.
New framework improves restless bandit policies for large numbers of arms.
problem Efficiently compute policies for large numbers of arms in restless bandit problems.
method Follow-the-Virtual-Advice framework, converting single-armed policies to N-armed policies.
result Achieves an O(1/\sqrt{N}) optimality gap in both discrete and continuous settings.
We study the restless bandit associated with an extremely simple scalar Kalman filter model in discrete time. Under certain assumptions, we prove that the problem is indexable in the sense that the Whittle index is a non-decreasing function of the relevant belief state. In spite of the long history of this problem, thi…
We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after T T T steps achieves O ~ ( T ) \tilde{O}(\sqrt{T}) O ~ ( T ) regret with respect to the best policy that knows the distributions of all arms. No ass…
Proposes a fair RMAB framework ensuring equal exposure to arms.
problem Fairness in RMABs where arms are not equally exposed.
method Defines merit of each arm based on stationary reward distribution and ensures equal exposure in proportion to merit.
result Achieves sublinear fairness regret of O ( T ln T ) O(\sqrt{T\ln T}) O ( T ln T ) in single pull case. DOPL learns from preference feedback to solve RMAB problems.
problem Learning optimal decisions in RMAB with limited reward information.
method Direct online preference learning (DOPL) for Pref-RMAB.
result DOPL achieves sublinear regret for RMAB with preference feedback.
Study best arm identification in restless Markov multi-armed bandits with state-dependent transitions.
problem Identify the best arm in a multi-armed bandit with time-varying states.
method Propose a sequential policy to select arms without knowing their exact TPMs.
result Upper and lower bounds on expected time to find the best arm match in a special case.
New RL algorithm tackles adversarial RMAB with unknown transitions and bandit feedback.
problem Learning in episodic RMAB with unknown transition functions and adversarial rewards.
method Developed a novel RL algorithm with a biased reward estimator and an index policy.
result Achieved i l d e O ( H T ) ilde{\mathcal{O}}(H\sqrt{T}) i l d e O ( H T ) regret bound for adversarial RMAB. New algorithm optimizes resource allocation in non-stationary networks.
problem Optimal resource allocation in non-stationary RMABs is computationally hard.
method Sliding-Window Online Whittle (SW-Whittle) policy for non-stationary transition kernels.
result Sub-linear dynamic regret achieved with unknown variation budget.
Bayesian bandits misspecification affects UX optimization, revealing new models.
problem Misspecification of value models in Bayesian bandits impacts UX optimization.
method Formulated UXO as a restless, sleeping bandit with unobserved confounders and optional stopping. Provided model extensions to address misspecifications.
result Common misspecifications lead to sub-optimal rewards, demonstrating overdispersion's effects on bandit performance.
Algorithm reduces regret in restless multi-armed bandits by adaptively sequencing arm choices.
problem Minimizing regret in restless multi-armed bandits with unknown dynamics.
method Adaptive Sequencing Rules (ASR) algorithm that selects arms in a consecutive manner.
result Achieves logarithmic regret order with time and finite-sample bound.
Neural Index Policy for multi-action bandits with heterogeneous budgets.
problem Real-world settings often involve multiple interventions with heterogeneous costs and constraints, breaking classical assumptions.
method Introduces a Neural Index Policy (NIP) that learns to assign budget-aware indices to arm-action pairs using a neural network and differentiable knapsack layer.
result Empirically achieves near-optimal performance while strictly enforcing heterogeneous budgets and scaling to hundreds of arms.
We obtain the conditions for the emergence of the swarm intelligence effect in an interactive game of restless multi-armed bandit (rMAB). A player competes with multiple agents. Each bandit has a payoff that changes with a probability p c p_{c} p c per round. The agents and player choose one of three options: (1) Exploit (a …
ContextWIN uses neural networks and reinforcement learning to optimize decisions in dynamic environments.
problem Optimizing decisions in dynamic, context-aware environments like recommendation systems.
method Integrates a mixture of experts within a reinforcement learning framework to compute context-specific weights for decision-making.
result Enhanced efficiency and accuracy in Whittle index computation for each arm in RMABs.
New algorithms reduce regret in non-stationary bandits with increasing payoffs.
problem Non-stationary bandits with monotonically increasing payoffs.
method R-ed-UCB for rested case and R-less-UCB for restless case.
result Regret bound of O ~ ( T 2 3 ) \widetilde{\mathcal{O}}(T^{\frac{2}{3}}) O ( T 3 2 ) under certain conditions. Paper tackles identifying an odd arm in a multi-armed bandit with restless Markov processes and trembling hand.
problem Identifying an odd arm in a multi-armed bandit with restless Markov processes and trembling hand.
method Derive asymptotic lower bound on expected time to identify the odd arm, stitch together parameterised solutions to MDPs.
result First known asymptotic lower bound on expected time to identify the odd arm, with vanishing error probability.
Paper tackles anomaly detection in restless Markov arms with unknown TPMs.
problem Detecting an anomalous arm in a multi-armed bandit with unknown transition probability matrices.
method Developed a policy based on the principle of certainty equivalence, achieving the lower bound arbitrarily closely under specific assumptions.
result Achieved the lower bound on expected time required to find the odd arm index, demonstrating the policy's effectiveness.
Optimal policy found for observing noisy time series.
problem Minimizing posterior variance plus observation costs in discrete-time Gaussian random walks.
method Developed a simple threshold-based policy and proved its optimality.
result Simple threshold policy is optimal for observing noisy time series.
New bandit model for healthcare intervention planning.
problem Maximizing patient health with limited monitoring resources.
method Developed Collapsing Bandits model and derived optimal policies.
result 3-order-of-magnitude speedup in algorithm performance.
In this paper we propose a flexible and efficient framework for handling multi-armed bandits, combining sequential Monte Carlo algorithms with hierarchical Bayesian modeling techniques. The framework naturally encompasses restless bandits, contextual bandits, and other bandit variants under a single inferential model. …
Algorithm improves resource allocation for food outreach to homeless.
problem Resource-constrained outreach for homeless individuals and food rescue.
method Thompson sampling with Markov chain recovery (via Stein variational gradient descent) for partially-observed episodic restless bandits.
result Significantly outperforms baselines in both organizations' problems.
Improved control approach for correlated bandits with better performance.
problem General multi-armed bandit problem with correlated elements.
method Introducing entropy regularisation to obtain a smooth asymptotic approximation of the value function, leading to a semi-index approximation of the optimal decision process.
result Performance of Asymptotic Randomised Control (ARC) algorithm compares favorably with other approaches.
Bayesian approach improves bandit decision-making in changing conditions.
problem Improving decision-making in non-stationary environments for multi-armed bandits.
method Proposes a variant of Thompson Sampling with discounting and an optimistic version.
result Derives the probability of picking sub-optimal arms and validates the utility of the algorithm.
New algorithm optimizes decision-making for complex systems with varying parameters.
problem Optimizing decisions in systems with varying parameters and heterogeneous restlessness.
method Model Predictive Control (MPC) approach with randomized rounding for heterogeneous RMABs.
result Achieves an O ( log N 1 / N ) O(\log N\sqrt{1/N}) O ( log N 1/ N ) optimality gap in infinite time average reward problems. Optimal policy for multi-armed multi-action bandits with unknown parameters.
problem Optimal sequential action selection for multi-armed multi-action bandits with unknown parameters.
method Occupancy-Measured-Reward Index Policy (OMRIP) and R(MA)^2B-UCB algorithm.
result Asymptotically optimal policy with sub-linear regret and low computational complexity.
Study on bandits with fading memory, improving regret bounds.
problem Stochastic multi-armed bandit problem with dependent samples.
method Developed a $\cC-$ Mix Improved UCB algorithm and analyzed regret bounds in two scenarios.
result Regret bounds similar to independent case in slow mixing scenario, with an additive term.
New model for bandit problem with linear rewards and side information.
problem Hidden Markovian bandit problem with linear rewards and side information.
method Presented a model and algorithm with regret analysis for the problem.
result Logarithmic regret achieved even in high-dimensional problems with structural side information.
New approach tackles non-Markovian behavior in maternal health programs.
problem Improving adherence and engagement in maternal and child healthcare programs.
method Extending RMABs to non-Markovian settings, using time-series forecasting and TARI policy.
result Significant increase in engagement and content listened compared to existing methods.
Method predicts future rewards from past actions in a linear Gaussian system.
problem Maximizing cumulative reward in a stochastic multi-armed bandit with linear Gaussian dynamics.
method Proposes a method using a modified Kalman filter to predict future rewards based on past rewards.
result Reward from any action can be used to predict another action's future reward.
We consider effort allocation in crowdsourcing, where we wish to assign labeling tasks to imperfect homogeneous crowd workers to maximize overall accuracy in a continuous-time Bayesian setting, subject to budget and time constraints. The Bayes-optimal policy for this problem is the solution to a partially observable Ma…
Bayesian reflex models AI learning like the autonomic nervous system.
problem Online learning in dynamic AI environments.
method Bayesian online algorithms with belief maintenance, sequential updating, and uncertainty-driven action balancing.
result Unified framework for adaptive AI learning.
Lyapunov-based analysis shows polynomial sample complexity for WCMDPs and RBs.
problem Learning in WCMDPs and RBs under a generative model.
method Lyapunov-based analysis framework.
result Near-optimal policies can be learned with polynomial complexity.
The paper uncovers the mathematical structure enabling value decomposition in multi-agent systems.
problem Theoretical justification for why value decomposition works effectively in multi-agent systems remains underexplored.
method The paper introduces the concept of Markov entanglement to measure the underlying structure and demonstrates how it can be used to bound the decomposition error.
result The widely-used class of index policies is weakly entangled and enjoys a sublinear O ( N ) \mathcal O(\sqrt{N}) O ( N ) scale of decomposition error for N N N -agent systems. New algorithms tackle RKHS bandits with reduced complexity and improved performance.
problem Adversarial and stochastic RKHS bandit problems with high computational complexity.
method Combining approximation theory with misspecified linear bandit methods.
result First general algorithm for adversarial RKHS bandit problem.
Unified formulation bridges adversarial and nonstationary bandits.
problem Handling time-varying reward distributions in multi-armed bandit problems.
method Unified oracle that switches between adversarial and nonstationary bandit oracles based on window size.
result Optimal regret achieved with matching lower bound.
Paper studies attacks on bandit algorithms and shows how attackers can manipulate data to hijack behavior.
problem Potential attacks on bandit algorithms can cause catastrophic loss in real-world applications.
method Proposes a framework of offline and online attacks on bandit algorithms using convex optimization and adaptive strategies.
result Attackers can force bandit algorithms to pull target arms with high probability by manipulating data.