New algorithm reduces neural net error in contextual bandits.
problem Neural contextual bandits with general activation functions.
method Proposed an efficient algorithm with sublinear regret bound.
result Demonstrated provably sublinear regret bound in finite regime.
Neural Thompson Sampling uses deep neural networks for contextual bandit problems.
problem Solving contextual multi-armed bandit problems.
method Adapts deep neural networks for exploration and exploitation in a novel posterior distribution.
result Guaranteed cumulative regret of O ( T 1 / 2 ) \mathcal{O}(T^{1/2}) O ( T 1/2 ) for bounded reward functions. Proposes EE-Net for neural exploration in contextual bandits.
problem Exploitation-Exploration tradeoff in contextual bandits.
method Uses two neural networks: Exploitation and Exploration, to learn reward function and adaptively explore.
result Achieves O ( T log T ) \mathcal{O}(\sqrt{T\log T}) O ( T log T ) regret and outperforms existing methods. We study the neural-linear bandit model for solving sequential decision-making problems with high dimensional side information. Neural-linear bandits leverage the representation power of deep neural networks and combine it with efficient exploration mechanisms, designed for linear contextual bandits, on top of the last…
Study shows efficient neural network approach for stochastic bandits.
problem Optimizing decisions in uncertain environments with neural network models.
method OFU-ReLU algorithm that balances exploration and exploitation, using a transformed feature space.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret guarantee for stochastic bandits with ReLU neural networks. Neural- σ 2 σ^2 σ 2 -LinearUCB improves regret in neural contextual bandits.
problem Balancing exploration and exploitation in neural contextual bandits.
method Proposes a variance-aware neural UCB algorithm using neural representations and an upper bound of reward noise variance.
result Oracle and practical versions of Neural- σ 2 σ^2 σ 2 -LinearUCB achieve better regret guarantees and performance. Unified framework for ensemble sampling in nonlinear contextual bandits with provable regret bounds.
problem Efficient exploration in nonlinear contextual bandits with unknown feature dimensions.
method Developed GLM-ES and Neural-ES for generalized linear and neural contextual bandits, respectively, using maximum likelihood estimation on randomly perturbed data.
result Unified high-probability frequentist regret bounds for GLM-ES and Neural-ES, matching state-of-the-art results.
NeurWIN learns Whittle indices for restless bandits using deep reinforcement learning.
problem Finding optimal solutions for restless bandits with complex transition kernels.
method NeurWIN uses a neural network to learn Whittle indices, leveraging the Whittle indices' mathematical properties.
result NeurWIN outperforms other RL algorithms in solving three recently studied restless bandit problems.
New algorithms use neural networks to minimize regret in contextual bandits.
problem Minimizing regret in sequential decisions with side information.
method Proposes NN-UCB and NTK-UCB algorithms using neural networks.
result Proves sublinear regret bounds for contextual bandits.
NK bandits improve performance on nonlinear tasks.
problem Improving performance on nonlinear sequential decision tasks.
method Proposed NK bandits using neural kernels to guide policies.
result NK bandits achieve state-of-the-art performance.
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear ba…
AGG-UCB uses neural networks to optimize group behaviors in contextual bandits.
problem Optimizing group behaviors in contextual bandits with mutual impacts.
method Introduces Arm Group Graph (AGG) and AGG-UCB algorithm using neural networks and graph neural networks.
result Achieves near-optimal regret bound with over-parameterized neural networks.
New algorithms for neural bandits learn from context and arm features.
problem Learning from context and arm features in combinatorial bandit problems.
method Proposed algorithms CN-UCB and CN-TS using deep neural networks.
result Achieved regret guarantees for the first time in combinatorial neural bandits.
Selective reinitialization improves adaptability of neural bandits in dynamic environments.
problem Loss of plasticity in neural bandits, leading to rigid neural network parameters.
method Selective Reinitialization (SeRe) framework that dynamically resets underutilized units.
result SeRe enhances adaptability of CNB algorithms, reducing cumulative regret in dynamic environments.
AdamCB optimizes neural network training by adaptively selecting samples.
problem Inefficient convergence due to unequal influence of different data samples.
method Integrates combinatorial bandit techniques into Adam to adaptively select samples.
result AdamCB achieves faster convergence and better performance than existing methods.
A new algorithm learns from raw feature vectors using deep neural networks and UCB for exploration.
problem Learning from raw feature vectors with unknown reward functions.
method Deep representation learning followed by UCB exploration in the last layer.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) finite-time regret. Study sparsity benefits in infinite feature contextual bandits.
problem Minimizing regret in infinite feature contextual bandits.
method Novel reduction to multi-armed bandits, Feel-Good Thompson Sampling algorithm.
result Regret bounds match lower bounds up to logarithmic factors, logarithmic dependence on effective features.
Survey and compare PAC-Bayes bounds for bandit problems.
problem Designing and evaluating bandit algorithms with strong performance guarantees.
method PAC-Bayes bounds applied to bandit problems.
result PAC-Bayes bounds useful for offline bandit algorithms, but loose for online algorithms.
Paper proposes efficient offline neural bandit method.
problem Offline policy learning with neural networks.
method Provable efficient offline contextual bandit with neural network function approximation.
result Method provably generalizes over unseen contexts.
Algorithm adapts to non-stationary rewards without prior knowledge.
problem Optimizing decisions in non-stationary environments without prior knowledge of changes.
method Optimization-based algorithm that restarts when non-stationarity is detected.
result Achieves tighter dynamic regret bound and is nearly minimax optimal.
Neural algorithms optimize arm selection with human preference feedback for complex reward functions.
problem Optimizing arm selection with noisy human preference feedback for complex, non-linear reward functions.
method Neural network to estimate reward function using preference feedback, upper confidence bound and Thompson sampling algorithms.
result Sub-linear regret guarantees for efficient arm selection in contextual dueling bandits.
New algorithms improve contextual bandits with neural networks and energy models.
problem Inefficient exploration in non-linear models for contextual bandits.
method Maximum entropy exploration using neural networks and energy models.
result Both techniques outperform standard algorithms, with energy models best overall.
Batched Neural Bandits reduces policy updates in sequential decision-making.
problem Sequential decision-making with batched policy changes.
method BatchNeuralUCB algorithm combining neural networks and optimism.
result Achieves similar regret as fully sequential version with fewer policy updates.
Graph neural Thompson Sampling improves online decision-making for graph data.
problem Online decision-making with graph-structured rewards.
method GNN-TS algorithm using GNN for mean reward estimation and graph neural tangent features for uncertainty.
result GNN-TS achieves a state-of-the-art regret bound of i l d e O ( ( i l d e d T ) 1 / 2 ) ilde{\mathcal{O}}(( ilde{d} T)^{1/2}) i l d e O (( i l d e d T ) 1/2 ) . Optimizes algorithms for non-concave bandit problems.
problem Optimizing algorithms for non-concave bandit problems.
method Unified zeroth-order optimization paradigm.
result Minimax-optimal algorithms in the dimension for low-rank generalized linear bandit problems.
A new approach learns to represent context for nonstationary bandits.
problem Nonstationary contextual bandits where patterns change over time.
method Combines recurrent neural networks with contextual linear bandit algorithm.
result Consistently outperforms handcrafted historical contexts and other methods.
New algorithms reduce regret in neural logistic bandits.
problem Learning unknown reward functions in neural networks.
method Introduced a Bernstein-type inequality for self-normalized vector-valued martingales.
result Regret bounds improved to O ~ ( d ~ κ T ) \widetilde{O}(\widetilde{d}\sqrt{κT}) O ( d κ T ) and O ~ ( d ~ T / κ ) \widetilde{O}(\widetilde{d}\sqrt{T/κ}) O ( d T / κ ) . We study the stochastic contextual bandit problem, where the reward is generated from an unknown function with additive noise. No assumption is made about the reward function other than boundedness. We propose a new algorithm, NeuralUCB, which leverages the representation power of deep neural networks and uses a neural…
This work improves online regression and contextual bandits using neural networks.
problem Improving online regression and contextual bandits using neural networks.
method Investigates neural networks for online regression, showing O ( log T ) \mathcal{O}(\log T) O ( log T ) regret for almost convex losses and KL loss. result Shows i l d e O ( K L ∗ + K ) ilde{\mathcal{O}}(\sqrt{KL^*} + K) i l d e O ( K L ∗ + K ) regret for NeuCB, outperforming existing algorithms. New algorithms reduce dueling bandits' regret with neural networks and efficient exploration.
problem Optimizing dueling bandits with neural networks for better performance.
method Combines shallow exploration strategies with neural networks for utility approximation, using iterative self-improvement and spectral analysis to reduce network width.
result Achieves sublinear regret of O ~ ( d ∑ t = 1 T σ t 2 + d T ) \widetilde{\mathcal{O}}(d\sqrt{\sum_{t=1}^{T} σ_t^2} + \sqrt{dT}) O ( d ∑ t = 1 T σ t 2 + d T ) . Paper proposes an efficient bandit-based algorithm for hyperparameter optimization.
problem Efficiently evaluating hyperparameters in deep learning models with large search spaces.
method Sub-Sampling (SS) algorithm combined with Bayesian Optimization (BOSS).
result Theoretical proof of optimality and empirical validation of superior performance.
New algorithm tackles non-linear utility in MNL bandits with i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret.
problem Sequential assortment selection with intricate user-item interactions.
method Upper Confidence Bound principle for non-linear parametric utility functions, including neural networks.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret bound for neural network-based utilities. NeuralRBMLE tackles explore-exploit trade-offs in contextual bandits with neural networks.
problem Stochastic contextual bandit problem with general bounded reward functions.
method Reward-biased maximum likelihood estimation with neural networks to enforce exploration.
result Both NeuralRBMLE variants achieve O ~ ( T ) \widetilde{\mathcal{O}}(\sqrt{T}) O ( T ) regret. BLISS optimizes GNN training by adaptively sampling nodes.
problem High computational costs in training GNNs on large graphs.
method Uses Bandit Layer Importance Sampling to dynamically select nodes.
result Improves GNN performance with reduced computational cost.
Study pure exploration in high-dimensional feature spaces using adaptive embeddings.
problem Overcoming the curse of dimensionality in pure exploration bandits.
method Adaptive embedding of feature representations into lower-dimensional spaces, carefully dealing with model misspecification.
result Sample complexity guarantees that depend on the effective dimension of feature spaces in kernel or neural representations.
Paper tackles domain adaptation for contextual bandits with sub-linear regret.
problem Adapting contextual bandit algorithms across domains with distribution shift.
method Learn a bandit model for the target domain using feedback from the source domain.
result Sub-linear regret bound maintained across domains.
Neural Index Policy for multi-action bandits with heterogeneous budgets.
problem Real-world settings often involve multiple interventions with heterogeneous costs and constraints, breaking classical assumptions.
method Introduces a Neural Index Policy (NIP) that learns to assign budget-aware indices to arm-action pairs using a neural network and differentiable knapsack layer.
result Empirically achieves near-optimal performance while strictly enforcing heterogeneous budgets and scaling to hundreds of arms.
A new exploration strategy for contextual bandits reduces regret and is computationally efficient.
problem Improving exploration in contextual bandits to reduce regret.
method Feature perturbation, injecting randomness directly into feature inputs.
result Achieves i l d e O ( d T ) ilde{\mathcal{O}}(d\sqrt{T}) i l d e O ( d T ) worst-case regret bound, surpassing existing methods. Ensemble++ uses shared-factor ensembles to scale Thompson Sampling for linear and nonlinear bandits.
problem Computational challenges in Thompson Sampling for large-scale or non-conjugate settings.
method Ensemble++ with shared-factor architecture and random linear combinations.
result Ensemble++ achieves comparable regret to exact Thompson Sampling with significantly smaller ensemble sizes.
New framework uses tree ensembles for contextual bandits.
problem Optimizing decisions in dynamic environments with contextual information.
method Adapts Upper Confidence Bound and Thompson Sampling to tree ensemble methods.
result Tree ensemble methods outperform traditional methods in regret minimization and runtime.
Stochastic Lipschitz bandit algorithms balance exploration and exploitation, and have been used for a variety of important task domains. In this paper, we present a framework for Lipschitz bandit methods that adaptively learns partitions of context- and arm-space. Due to this flexibility, the algorithm is able to effic…
We introduce a new and completely online contextual bandit algorithm called Gated Linear Contextual Bandits (GLCB). This algorithm is based on Gated Linear Networks (GLNs), a recently introduced deep learning architecture with properties well-suited to the online setting. Leveraging data-dependent gating properties of …
IDS improves reinforcement learning with contextual information.
problem Optimizing IDS for contextual reinforcement learning.
method Investigated contextual bandit problems and proposed a computationally-efficient IDS.
result Contextual IDS outperforms conditional IDS by considering future contexts.
Contextual multi-armed bandit problems arise frequently in important industrial applications. Existing solutions model the context either linearly, which enables uncertainty driven (principled) exploration, or non-linearly, by using epsilon-greedy exploration policies. Here we present a deep learning framework for cont…
Study identifies methods for handling sparse data in web settings with contextual bandits.
problem Handling sparse data in web settings with contextual bandits.
method Identified and categorized methods for addressing sparse data issues.
result Updated understanding of sparse data problems using contextual bandits in web settings.
Master-slave architecture tackles combinatorial multi-armed bandits with diversity constraints.
problem Solving top- K K K combinatorial multi-armed bandits with non-linear feedback and diversity constraints. method Master-slave architecture with six slave models, teacher learning, and policy co-training.
result Significantly outperforms existing algorithms in synthetic and real datasets.
We study two randomized algorithms for generalized linear bandits. The first, GLM-TSL, samples a generalized linear model (GLM) from the Laplace approximation to the posterior distribution. The second, GLM-FPL, fits a GLM to a randomly perturbed history of past rewards. We analyze both algorithms and derive $\tilde{O}(…
The paper explores sampling problems and shows minimal exploration is needed.
problem Exploration-exploitation trade-off in sampling.
method Systematic definition of regret, proposal of a simple algorithm.
result Near-optimal regret bounds achieved with minimal exploration.