SS-SARSA tackles recovering bandits by treating rounds as states.
problem Real-world rewards vary with the elapsed rounds since last pull.
method Proposes SS-SARSA, a new RL algorithm treating rounds as states.
result Achieves efficient learning with asymptotic convergence to optimal policy.
We study the recovering bandits problem, a variant of the stochastic multi-armed bandit problem where the expected reward of each arm varies according to some unknown function of the time since the arm was last played. While being a natural extension of the classical bandit problem that arises in many real-world settin…
New policy tackles evolving externalities in contextual bandits.
problem Difficulty in recovering from wrong decisions over time.
method Rejection-based policy to achieve low regret.
result Low regret achieved regardless of reward matrix structure.
New algorithm improves learning efficiency in multi-task contextual bandits.
problem Improving learning efficiency in multi-task contextual bandits.
method Alternating projected gradient descent (GD) and minimization estimator for low-rank feature matrix recovery.
result Proved regret bound for multi-task learning algorithm.
New algorithm reduces multi-agent bandit regret by sharing data.
problem Designing efficient collaboration between multi-agent linear bandits.
method Bandit Adaptive Sample Sharing (BASS) algorithm, without assumptions on bandit parameters structure.
result Validated through theoretical analysis and empirical evaluations, BASS outperforms current state-of-the-art.
We analyze the K K K -armed bandit problem where the reward for each arm is a noisy realization based on an observed context under mild nonparametric assumptions. We attain tight results for top-arm identification and a sublinear regret of O ~ ( T 1 + D 2 + D ) \widetilde{O}\Big(T^{\frac{1+D}{2+D}}\Big) O ( T 2 + D 1 + D ) , where D D D is the context dimension, f…
We study a nonparametric contextual bandit problem where the expected reward functions belong to a Hölder class with smoothness parameter β β β . We show how this interpolates between two extremes that were previously studied in isolation: non-differentiable bandits ( β ≤ 1 β\leq1 β ≤ 1 ), where rate-optimal regret is achieved by run…
New policy for non-stationary multi-armed bandits with recovering rewards.
problem Non-stationary multi-armed bandits with recovering rewards.
method Purely Periodic Policies integrating upper confidence bound procedure.
result Approximately O ~ ( N T ) \widetilde{\mathcal O}(N\sqrt{T}) O ( N T ) regret against the offline benchmark. Paper addresses federated contextual bandits with encryption.
problem Building contextual bandits with vertically distributed data.
method Design of O3M encryption scheme for LinUCB and LinTS.
result Proposed protocols achieve good performance and privacy.
RONM method reduces regret in stochastic convex bandits with decreasing noise.
problem Stochastic convex bandit problem with decreasing noise.
method Regularized Online Newton Method (RONM) based on Online Newton Method (ONM).
result RONM achieves polylogarithmic regret in time horizon n.
Efficient algorithms for contextual bandits with smooth regret in continuous action spaces.
problem Efficient learning in large or continuous action spaces.
method Smooth regret notion and efficient algorithms for general function approximation.
result Statistically and computationally efficient algorithms for contextual bandits with smooth regret.
New bandit algorithm for non-i.i.d. noise, improving standard rates.
problem Linear stochastic bandit with non-i.i.d. observation noise.
method Developed new confidence sequences and an algorithm based on optimism in uncertainty.
result Regret bounds for the new algorithm, showing recovery of standard rates up to a factor of the mixing time.
DPPS uses DP priors for Bayesian non-parametric multi-arm bandits.
problem Optimizing multi-arm bandit environments with prior beliefs.
method Bayesian non-parametric algorithm based on Dirichlet Process priors.
result DPPS provides principled incorporation of prior beliefs and is optimal in Bayesian regret setup.
Paper refines PAC-Bayes bounds for bandit problems.
problem Improving probabilistic bounds for off-policy learning.
method Optimizes PAC-Bayesian bounds using a new parameter optimization approach.
result Provides two parameter-free PAC-Bayes bounds that nearly match optimal rates.
New algorithms for model selection in linear contextual bandits without feature diversity conditions.
problem Model selection in linear contextual bandits without feature diversity conditions.
method Data-adaptive algorithms that provide model selection guarantees without feature diversity conditions.
result O(d^α T^{1-α}) model selection guarantees with no feature diversity conditions.
We propose R a n d U C B \tt RandUCB RandUCB , a bandit strategy that builds on theoretically derived confidence intervals similar to upper confidence bound (UCB) algorithms, but akin to Thompson sampling (TS), it uses randomization to trade off exploration and exploitation. In the K K K -armed bandit setting, we show that there are infinitel…
This work explores adaptive strategies for multi-armed bandits with causal structure, achieving optimal regret bounds.
problem Adapting to causal structure in multi-armed bandits with additional observed variables.
method Reduction to linear bandits and establishment of Pareto optimal frontier of adaptive rates.
result Established upper and lower bounds on adaptive rates, resolving open questions.
New algorithm for collaborative bandit learning reduces sample complexity and regret.
problem Optimal arm identification in a multi-agent bandit model with communication.
method Phased elimination with data-dependent sampling schemes.
result Near-optimal algorithm for pure exploration in collaborative bandit learning.
We address online linear optimization problems when the possible actions of the decision maker are represented by binary vectors. The regret of the decision maker is the difference between her realized loss and the best loss she would have achieved by picking, in hindsight, the best possible action. Our goal is to unde…
Unified framework for analyzing online convex optimization across various settings.
problem Analyzing online convex optimization in different settings and feedback types.
method Unified framework allowing systematic proposal and analysis of meta-algorithms.
result Comparable regret bounds for various feedback types and adversary types.
Information-theoretic Bayesian regret bounds of Russo and Van Roy capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among o…
DE is a new exploration method that limits resource usage based on expected improvement and surprise.
problem Limited exploration in large action spaces when resources are scarce.
method Delight-gated exploration (DE) that limits exploration actions based on a gate price set by the product of expected improvement and surprise.
result DE outperforms ε \varepsilon ε -greedy and Thompson Sampling in terms of regret across various bandit and MDP settings. New algorithm for linear bandits tackles Optimal Transport problems.
problem Optimal Transport problems not covered by traditional linear bandits.
method Embed actions into a Hilbertian subspace, penalize optimism, use least-squares estimation.
result Achieves same regret bounds as OFUL but interpolates between i l d e O ( T ) ilde{\mathcal O}(\sqrt{T}) i l d e O ( T ) and O ( T ) {\mathcal O}(T) O ( T ) . Develops model selection for bandits balancing adversarial and stochastic guarantees.
problem Model selection in bandit scenarios with simultaneous adversarial and stochastic high-probability regret.
method Nested policy classes, balanced candidate regret bounds, mis-specification tests.
result Best of both world guarantees in linear bandits with simultaneous adversarial and stochastic environments.
Algorithm identifies Pareto set in bandits with contaminated feedback.
problem Identifying Pareto set in multi-objective bandits with adversarial contamination.
method Sample median-based multi-objective adaptive elimination algorithm.
result Sample complexity bound that depends on contamination probability.
New method for linear bandits with unknown sparsity, improving sparse regret bounds.
problem Sparse regret bounds for unknown sparsity and adversarial action sets.
method Combines online to confidence set conversions with randomized model selection over nested confidence sets.
result First sparse regret bounds for unknown sparsity and adversarial action sets.
Algorithm tackles adaptive discretization in adversarial Lipschitz bandits for dynamic pricing and auctions.
problem Adaptive discretization in adversarial Lipschitz bandits.
method Adversarial Zooming algorithm for adaptive discretization.
result First algorithm for adversarial Lipschitz bandits with instance-dependent regret bounds.
New algorithm for multi-fidelity bandits reduces costs and improves regret.
problem Optimizing decisions with varying costs and accuracy in multi-fidelity bandits.
method Cost complexity bounds, algorithmic framework, elimination-based algorithm.
result New regret definition and matching upper and lower bounds for elimination-based algorithm.
Unified framework controls false discovery rate in bandit multiple testing.
problem Designing adaptive algorithms to identify true discoveries in multiple hypothesis testing.
method Unified modular framework using e-processes for FDR control in arbitrary settings.
result Unified framework ensures FDR control for dependent and simultaneous arm queries.
New algorithm for recommending best arms with aggregated feedback.
problem Finding the best arm under aggregated feedback when precise rewards are unavailable.
method Gaussian Process Optimistic Optimisation (GPOO) algorithm with adaptive tree construction.
result The proposed algorithm achieves new simple regret bounds with aggregated feedback.
Unified framework for corruption-robust linear bandits with optimal gap-dependent misspecification bounds.
problem Effective learning in linear bandits with corrupted rewards across different corruption models.
method Unified framework for analyzing strong and weak corruption, connection to gap-dependent misspecification, and specialized algorithm.
result Optimal bounds for gap-dependent misspecification in linear bandits.
New algorithm identifies best target arm with known additive relationship between source and target MAB instances.
problem Identifying the best arm in a target MAB instance when only source arms can be pulled and there's a known additive relationship between the two.
method Proposes an LUCB-style algorithm to identify an ε-optimal target arm with high probability.
result Theoretical analysis highlights aspects of the transfer learning problem and recovers the LUCB algorithm for single domain BAI as a special case.
Adaptive algorithm selects models for stochastic linear bandits based on problem complexity.
problem Model selection for stochastic linear bandits with unknown problem complexity.
method Adaptive phase-based algorithm that adapts to true problem complexity.
result Achieves regret scaling of O ( ∥ θ ∗ ∥ T ) O(\|θ^*\|\sqrt{T}) O ( ∥ θ ∗ ∥ T ) for K K K armed mixture bandits and O ( d ∗ T ) O(d^*\sqrt{T}) O ( d ∗ T ) for standard linear bandits. In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in a…
Improved regret bounds for linear bandits with heavy-tailed rewards.
problem Stochastic linear bandits with heavy-tailed rewards.
method Elimination-based algorithm guided by experimental design.
result Regret bound of \(\tilde{\mathcal{O}}(d^\frac{1+3ε}{2(1+ε)} T^\frac{1}{1+ε})\) for \(ε\in (0,1)\).
Observer learns optimal policy from learner's actions without rewards.
problem Learning optimal policy from non-rewarded actions of a non-stationary learner.
method Two-Phase Suffix Imitation framework.
result Observer achieves convergence rate of O ~ ( 1 / N ) \tilde O(1/\sqrt{N}) O ~ ( 1/ N ) . Develops a method to tackle high-dimensional linear bandits with knapsacks using online sparse estimation and dual variables.
problem High-dimensional linear bandits with knapsacks.
method Online hard thresholding algorithm for sparse estimation, integrated with primal-dual scheme.
result Achieves sub-linear regret that scales logarithmically with feature dimension, improving on prior work.
We formulate the problem of sampling and recovering clustered graph signal as a multi-armed bandit (MAB) problem. This formulation lends naturally to learning sampling strategies using the well-known gradient MAB algorithm. In particular, the sampling strategy is represented as a probability distribution over the indiv…
New methods combat data poisoning attacks in bandit algorithms using limited verification.
problem Data poisoning attacks on bandit algorithms, especially in the UCB and ETC types.
method Verification-based mechanisms to restore optimal regret with limited verifications.
result A simple modified ETC type bandit algorithm can restore optimal regret with O ( log T ) O(\log T) O ( log T ) verifications. We address the online linear optimization problem when the actions of the forecaster are represented by binary vectors. Our goal is to understand the magnitude of the minimax regret for the worst possible set of actions. We study the problem under three different assumptions for the feedback: full information, and the …
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
New study reveals a polynomial penalty for adapting to unknown margin parameters in batched nonparametric bandits.
problem Adapting to an unknown margin parameter in batched nonparametric bandits.
method Introduces the regret inflation criterion and develops RoBIN algorithm to achieve optimal regret inflation.
result The optimal regret inflation grows polynomially with the horizon T, characterized by a convex optimization problem.
Algorithm clusters items by sequentially selecting features, minimizing observations.
problem Clustering items based on bandit feedback with many features.
method Sequential Halving algorithm for feature selection.
result Accurate recovery of item partition with minimal observations.
New approach turns optimal stationary RL into non-stationary RL without prior knowledge.
problem Optimal RL in non-stationary environments without prior knowledge of non-stationarity.
method Black-box reduction of optimal stationary RL algorithms to non-stationary RL.
result Achieves optimal dynamic regret bounds in various RL settings.
New algorithm tackles heterogeneous curvature in online convex optimization.
problem Adversarial bandit convex optimization with varying curvature.
method Developed an adaptive algorithm that learns curvature on the fly.
result Achieves optimal regret bounds even with heterogeneous curvature.
Federated learning for combinatorial multi-agent bandits reduces regret and speeds up with fewer communications.
problem Online combinatorial optimization with noisy feedback and cooperation.
method Transforms offline algorithms into online multi-agent algorithms with sublinear regret and communication efficiency.
result Achieves sublinear regret and linear speedup with more agents, communication-efficient.
Study experiment planning with function approximation in contextual bandit problems.
problem Designing effective data collection strategies in settings with limited rewards.
method Proposes two experiment planning strategies compatible with function approximation.
result Eluder planning and sampling procedure achieves optimality guarantees.
New findings show optimization is crucial for OPL in large action spaces.
problem Challenges in optimizing policies for large action spaces in offline contextual bandits.
method Weighed log-likelihood objectives and estimator-aware policy parametrization.
result Simple weighted log-likelihood objectives enjoy better optimization properties and recover competitive policies.