The paper develops a Hoeffding inequality for Markov chains and applies it to bandit problems.
problem Developing a Hoeffding inequality for Markov chains and applying it to bandit problems.
method Developed a Hoeffding inequality for the partial sums of an irreducible Markov chain on a finite state space.
result Demonstrated the inequality's effectiveness in identifying approximately best Markovian arms and minimizing regret in Markovian bandits.
Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.
problem Optimizing a policy gradient algorithm for multi-arm bandit problems with variable learning rates.
method Applied Foster-Lyapunov techniques to analyze a Markov chain formed by the state of the algorithm.
result The policy gradient algorithm converges to the optimal arm with logarithmic or poly-logarithmic regret.
Study of multi-armed bandits with state-switching rewards using Markov models.
problem Multi-armed bandit problem with state-switching rewards.
method Spectral method-of-moments estimations for hidden Markov models, belief error control, upper-confidence-bound methods.
result Upper bound of O ( T 2 / 3 log T ) O(T^{2/3}\sqrt{\log T}) O ( T 2/3 log T ) for the learning algorithm performance. We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after T T T steps achieves O ~ ( T ) \tilde{O}(\sqrt{T}) O ~ ( T ) regret with respect to the best policy that knows the distributions of all arms. No ass…
A new recommendation system model tackles unreliable user behavior.
problem Creating effective recommendation systems in the presence of unreliable user behavior.
method A novel modification of Multi-Armed Bandits with an unreliable intermediate.
result Proved fundamental theorems and developed an Explore-Commit algorithm close to optimal performance.
Algorithm for online decision making with unknown dynamics and aggregate feedback.
problem Online decision making with unknown dynamics and aggregate bandit feedback.
method Developed an algorithm based on online mirror descent with a self-concordant barrier regularization and an increasing learning rate schedule.
result Achieved O ( K ) O(\sqrt{K}) O ( K ) regret for the online Markov Decision Process with K K K episodes. Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
A new MDP with Bandits approach for sequential decision making in linear-flow scenarios.
problem Sequential decision making with limited feedback in a linear-flow context.
method Formulated as an MDP with Bandits, using Thompson sampling for action selection and exact dynamic programming for allocation.
result The proposed MDP with Bandits algorithm outperforms other methods in sequential decision making.
Study best arm identification in restless bandits with unknown TPMs.
problem Identify the best arm with fixed confidence in restless bandits with unknown TPMs.
method Proposed a policy for best arm identification and proved its expected stopping time matches the lower bound.
result The state-action visitation proportions match the optimal proportions under any asymptotically optimal policy.
Study best arm identification in restless Markov multi-armed bandits with state-dependent transitions.
problem Identify the best arm in a multi-armed bandit with time-varying states.
method Propose a sequential policy to select arms without knowing their exact TPMs.
result Upper and lower bounds on expected time to find the best arm match in a special case.
ARL uses queries to learn rewards, focusing on cost vs. reward value.
problem How to efficiently use queries to learn rewards in reinforcement learning.
method Proposed and evaluated heuristic approaches for ARL in multi-armed bandits and MDPs.
result Challenging aspects of ARL highlighted, including intractability of value computation.
Adaptive KL-UCB algorithm for Markov and i.i.d. rewards.
problem Regret minimization for Markovian and i.i.d. rewards in MAB problems.
method Identifies Markovian vs. i.i.d. rewards, switches between KL-UCB variants.
result Logarithmic regret for both i.i.d. and Markovian settings.
FTPL method shows near-optimal regret bounds for AMDPs with bandit feedback.
problem Minimizing regret in AMDPs with adversarial losses and bandit feedback.
method Follow-the-Perturbed-Leader (FTPL) method for AMDPs.
result FTPL achieves near-optimal regret bounds for AMDPs with bandit feedback.
A new algorithm for cryo-EM data collection that balances reward and latency.
problem Optimizing data collection in cryo-EM experiments with action delays.
method Latency-aware contextual bandit framework and COAF algorithm.
result The COAF algorithm efficiently maximizes reward over time in cryo-EM experiments.
Study optimal adaptive allocation for multi-armed bandits with Markovian rewards.
problem Optimal adaptive allocation for multi-armed bandits with Markovian rewards.
method Round-robin Kullback-Leibler upper confidence bounds for optimal adaptive allocation.
result Logarithmic dependence of regret on time horizon, asymptotically optimal.
New strategy identifies best Markovian arm with fixed confidence.
problem Identifying the best arm in Markovian bandit models with fixed confidence.
method Analyzed the Track-and-Stop strategy and derived a concentration inequality for Markov chains.
result The Track-and-Stop strategy is at most a factor of four apart from the lower bound for asymptotic performance.
New model for bandit problem with linear rewards and side information.
problem Hidden Markovian bandit problem with linear rewards and side information.
method Presented a model and algorithm with regret analysis for the problem.
result Logarithmic regret achieved even in high-dimensional problems with structural side information.
Paper tackles identifying an odd arm in a multi-armed bandit with restless Markov processes and trembling hand.
problem Identifying an odd arm in a multi-armed bandit with restless Markov processes and trembling hand.
method Derive asymptotic lower bound on expected time to identify the odd arm, stitch together parameterised solutions to MDPs.
result First known asymptotic lower bound on expected time to identify the odd arm, with vanishing error probability.
Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.
problem Online learning in finite-horizon episodic MDPs with aggregate bandit feedback.
method Best-of-both-worlds (BOBW) algorithms using FTRL over occupancy measures, self-bounding techniques, and new loss estimators.
result First BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback achieving O ( log T ) O(\log T) O ( log T ) regret in stochastic and O ( T ) {O}(\sqrt{T}) O ( T ) regret in adversarial settings. New algorithm matches best regret bound for tabular Contextual Bandit problems.
problem Learning from observations in uncertain environments.
method A minor variant of a reinforcement learning algorithm for MDPs.
result The algorithm matches the best possible regret bound i l d e O ( S A T ) ilde O (\sqrt{SAT}) i l d e O ( S A T ) for tabular Contextual Bandit problems. Improved Thompson Sampling for high-dimensional sparse bandits.
problem Stochastic linear contextual bandits with high-dimensional features.
method Thompson Sampling with spike-and-slab priors and variational inference.
result Nearly optimal upper bound on expected cumulative regret.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
NeurWIN learns Whittle indices for restless bandits using deep reinforcement learning.
problem Finding optimal solutions for restless bandits with complex transition kernels.
method NeurWIN uses a neural network to learn Whittle indices, leveraging the Whittle indices' mathematical properties.
result NeurWIN outperforms other RL algorithms in solving three recently studied restless bandit problems.
The paper tackles Nash-regret minimization in congestion games with bandit feedback.
problem Minimizing Nash-regret in congestion games with bandit feedback.
method Proposes centralized and decentralized algorithms for congestion games with bandit feedback, and a centralized algorithm for Markov congestion games.
result Sample complexity depends polynomially on the number of players and facilities, not the size of the action set.
Two algorithms improve online reinforcement learning in adversarial linear MDPs with bandit feedback.
problem Online reinforcement learning in linear MDPs with adversarial losses and bandit feedback.
method Two algorithms: one computationally inefficient with $\widetilde{\mathcal{O}}\left(\sqrt{K}
ight)$ regret, and one computationally efficient with $\widetilde{\mathcal{O}}\left(K^{\frac{3}{4}}
ight)$ regret.
result Achieved improved regret performance compared to existing approaches.
New approach for online learning with adaptive adversaries, simpler and more effective.
problem Online learning with adaptive adversaries, especially in bandits and MDPs.
method Uses standard unbiased estimators and a simple increasing learning rate schedule, aided by logarithmically homogeneous self-concordant barriers and strengthened Freedman's inequality.
result First high-probability regret bounds for adversarial bandits and MDPs, resolving open problems.
Paper tackles anomaly detection in restless Markov arms with unknown TPMs.
problem Detecting an anomalous arm in a multi-armed bandit with unknown transition probability matrices.
method Developed a policy based on the principle of certainty equivalence, achieving the lower bound arbitrarily closely under specific assumptions.
result Achieved the lower bound on expected time required to find the odd arm index, demonstrating the policy's effectiveness.
LMC-TS uses MCMC for efficient posterior sampling in contextual bandits.
problem Efficiency of Thompson sampling for high-dimensional contextual bandits.
method Langevin Monte Carlo for direct posterior sampling.
result LMC-TS achieves sublinear regret bound for linear contextual bandits.
The design of personalized incentives or recommendations to improve user engagement is gaining prominence as digital platform providers continually emerge. We propose a multi-armed bandit framework for matching incentives to users, whose preferences are unknown a priori and evolving dynamically in time, in a resource c…
New algorithm reduces regret in RL with adversarial corruption.
problem Adversarial corruption in reinforcement learning.
method Uncertainty-weighted least-squares regression and weighted uncertainty estimator.
result Achieves regret of i l d e O ( T + ζ ) ilde{O}(\sqrt{T} + ζ) i l d e O ( T + ζ ) for contextual bandits. Improved Thompson sampling for bandits with Langevin algorithms.
problem Thompson sampling's computational inefficiency in generating posterior samples.
method Developed Langevin algorithms for approximate sampling with posterior concentration guarantees.
result Logarithmic regret with constant computational complexity.
BIS uses bandits to efficiently sample from expensive-to-evaluate densities.
problem Sampling from computationally expensive target densities.
method Sequential selection through multi-armed bandits, optimizing sample set directly.
result BIS achieves accurate sampling with fewer evaluations than adaptive methods.
New RL method tackles dynamic MDPs with evolving rewards and states.
problem Dynamic MDPs with evolving rewards and states.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).
result Achieves dynamic regret bound for non-stationary MDPs.
The paper extends Hoeffding's inequality for Markov chains using a generalized concentrability condition.
problem Applying Hoeffding's inequality to non-ergodic Markov chains.
method Integrates generalized concentrability condition via IPM to extend traditional hypotheses.
result Demonstrates utility in machine learning applications such as empirical risk minimization and bandits.
We consider online learning for minimizing regret in unknown, episodic Markov decision processes (MDPs) with continuous states and actions. We develop variants of the UCRL and posterior sampling algorithms that employ nonparametric Gaussian process priors to generalize across the state and action spaces. When the trans…
Narendra-Shapiro (NS) algorithms are bandit-type algorithms that have been introduced in the sixties (with a view to applications in Psychology or learning automata), whose convergence has been intensively studied in the stochastic algorithm literature. In this paper, we adress the following question: are the Narendra-…
A planning approach learns skills from interactions, balancing exploration and exploitation.
problem Learning robust high-level skills in noisy environments with unknown pre-conditions.
method Formulates skills as high-level policies, learns plans via bandit problems, balances exploration and exploitation.
result A planner capable of learning robust high-level skills in high-dimensional state spaces.
Paper tackles non-monotonic resource utilization in sequential decision-making.
problem Sequential decision-making under uncertainty with resource constraints.
method Introduces a new MDP policy with constant regret against LP relaxation.
result Develops a learning algorithm with logarithmic regret for unknown outcome distributions.
Efficient algorithm for learning MDPs with unknown transitions and bandit feedback.
problem Learning in episodic finite-horizon MDPs with unknown transitions and bandit feedback.
method Proposes an efficient algorithm with i l d e O ( L ∣ X ∣ ∣ A ∣ T ) \mathcal{ ilde{O}}(L|X|\sqrt{|A|T}) i l d e O ( L ∣ X ∣ ∣ A ∣ T ) regret. result Achieves i l d e O ( T ) \mathcal{ ilde{O}}(\sqrt{T}) i l d e O ( T ) regret, matching previous work with full-information feedback. New algorithm achieves best-of-both-worlds performance in various online learning settings.
problem Achieving optimal performance in both adversarial and stochastic online learning settings.
method General reduction from best-of-both worlds to FTRL and OMD algorithms.
result Transformed existing algorithms into new ones with best-of-both-worlds guarantees.
A new method for risk-averse decision-making in Markov processes with improved regret bounds.
problem Risk-averse decision-making in Markov processes.
method Introduces mini-batch measures and multipattern risk-averse problems in a feature-based Q Q Q -learning method. result Proves a high-probability regret bound of O ( H 2 N H K ) \mathcal{O}\big(H^2 N^H \sqrt{ K}\big) O ( H 2 N H K ) for the Q Q Q -learning method. Markov chain Monte Carlo (MCMC) algorithms are widely used to sample from complicated distributions, especially to sample from the posterior distribution in Bayesian inference. However, MCMC is not directly applicable when facing the doubly intractable problem. In this paper, we discussed and compared two existing solu…
New bandit model for healthcare intervention planning.
problem Maximizing patient health with limited monitoring resources.
method Developed Collapsing Bandits model and derived optimal policies.
result 3-order-of-magnitude speedup in algorithm performance.
We introduce Parameterized Exploration (PE), a simple family of methods for model-based tuning of the exploration schedule in sequential decision problems. Unlike common heuristics for exploration, our method accounts for the time horizon of the decision problem as well as the agent's current state of knowledge of the …
New algorithms for learning MDPs with linear approximations in infinite-horizon settings.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method Optimism principle, adversarial linear bandits, Natural Policy Gradient.
result Efficient algorithms with optimal or near-optimal regret bounds.
New algorithm for online learning in episodic MDPs with convex objectives.
problem Online episodic convex reinforcement learning.
method Online mirror descent algorithm with varying constraint sets and exploration bonus.
result Near-optimal regret bounds for online CURL without prior knowledge of transition function.
New RL algorithm tackles adversarial RMAB with unknown transitions and bandit feedback.
problem Learning in episodic RMAB with unknown transition functions and adversarial rewards.
method Developed a novel RL algorithm with a biased reward estimator and an index policy.
result Achieved i l d e O ( H T ) ilde{\mathcal{O}}(H\sqrt{T}) i l d e O ( H T ) regret bound for adversarial RMAB. Algorithm for online learning in MDPs with linear function approximation and bandit feedback.
problem Online learning in MDPs with changing reward functions and limited feedback.
method Developed MDP-LinExp3 algorithm with theoretical guarantees.
result Proved regret bounds for MDP-LinExp3 algorithm.