arXiv research

This paper improves Bayesian optimization methods with tighter regret bounds and practical solutions.

Improved algorithms for stochastic linear bandits using tighter confidence sequences.

We study reinforcement learning in non-episodic factored Markov decision processes (FMDPs). We propose two near-optimal and oracle-efficient algorithms for FMDPs. Assuming oracle access to an FMDP planner, they enjoy a Bayesian and a frequentist regret bound respectively, both of which reduce to the near-optimal bound …

RAVEN-UCB addresses non-stationary MAB problems with tighter regret bounds.

New algorithm reduces regret in GLM bandits with tighter bounds.

We investigate online convex optimization in changing environments, and choose the adaptive regret as the performance measure. The goal is to achieve a small regret over every interval so that the comparator is allowed to change over time. Different from previous works that only utilize the convexity condition, this pa…

Improved regret bounds for DP-KLUCB and DP-IMED in Bernoulli bandits.

Strong worst-case performance bounds for episodic reinforcement learning exist but fortunately in practice RL algorithms perform much better than such bounds would predict. Algorithms and theory that provide strong problem-dependent bounds could help illuminate the key features of what makes a RL problem hard and reduc…

Optimal simple regret bound for Gaussian Process bandits.

This paper considers the stability of online learning algorithms and its implications for learnability (bounded regret). We introduce a novel quantity called {\em forward regret} that intuitively measures how good an online learning algorithm is if it is allowed a one-step look-ahead into the future. We show that given…

New algorithm optimizes Hölder smooth functions in RKHS with tighter regret bounds.

Two algorithms tackle heavy-tailed rewards in reinforcement learning with linear function approximation.

PopArt efficiently solves sparse linear bandits with tighter recovery guarantees.

Paper tackles non-stationary kernelized bandits with near-optimal algorithm.

We present a generalization of the adversarial linear bandits framework, where the underlying losses are kernel functions (with an associated reproducing kernel Hilbert space) rather than linear functions. We study a version of the exponential weights algorithm and bound its regret in this setting. Under conditions on …

Improved Thompson Sampling algorithms for bandits with tighter regret bounds.

Improved reinforcement learning for episodes with varying action sets.

New algorithms exploit mean bounds to improve bandit problem performance.

A new framework tightens risk measure confidence bounds.

We consider the problem of learning in episodic finite-horizon Markov decision processes with an unknown transition function, bandit feedback, and adversarial losses. We propose an efficient algorithm that achieves $\mathcal{\tilde{O}}(L|X|\sqrt{|A|T})$ regret with high probability, where $L$ is the horizon, $|X|$ is t…

The problem of bipartite ranking, where instances are labeled positive or negative and the goal is to learn a scoring function that minimizes the probability of mis-ranking a pair of positive and negative instances (or equivalently, that maximizes the area under the ROC curve), has been widely studied in recent years. …

New algorithms reduce risk in reinforcement learning with provable regret bounds.

A new algorithm tackles adversarial linear contextual bandits using kernelized loss functions.

New algorithm reduces semi-bandit regret using covariance estimates.

Counterfactual regret minimization (CFR) is the most popular algorithm on solving two-player zero-sum extensive games with imperfect information and achieves state-of-the-art performance in practice. However, the performance of CFR is not fully understood, since empirical results on the regret are much better than the …

Improved online confidence bounds for multinomial logistic models in bandits.

Motivated by the pressing need for efficient optimization in online recommender systems, we revisit the cascading bandit model proposed by Kveton et al. (2015). While Thompson sampling (TS) algorithms have been shown to be empirically superior to Upper Confidence Bound (UCB) algorithms for cascading bandits, theoretica…

This work improves regret minimization for logistic bandits by reducing dependence on a large constant.

New algorithms reduce dynamic regret for convex and smooth functions in non-stationary environments.

We study the problem of switching-constrained online convex optimization (OCO), where the player has a limited number of opportunities to change her action. While the discrete analog of this online learning task has been studied extensively, previous work in the continuous setting has neither established the minimax ra…

We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations…

New algorithm uses control variates to improve multi-armed bandit performance.

We investigate the piecewise-stationary combinatorial semi-bandit problem. Compared to the original combinatorial semi-bandit problem, our setting assumes the reward distributions of base arms may change in a piecewise-stationary manner at unknown time steps. We propose an algorithm, \texttt{GLR-CUCB}, which incorporat…

New method tracks shifts in infinite-armed bandits without prior knowledge.

Algorithm adapts to non-stationary rewards without prior knowledge.

Thompson Sampling remains differentially private with minimal modifications.

GP-UCB resolves sublinear regret for kernelized bandits.

Paper optimizes learning MDPs with adversarial rewards using linear models.

We present methods for online linear optimization that take advantage of benign (as opposed to worst-case) sequences. Specifically if the sequence encountered by the learner is described well by a known "predictable process", the algorithms presented enjoy tighter bounds as compared to the typical worst case bounds. Ad…

Algorithm finds Nash equilibria in complex games with function approximation.

Paper tightens PAC-Bayes bounds using coin-betting for better estimates.

Improved online learning with time-varying constraints for complex domains.

The stochastic linear bandit problem proceeds in rounds where at each round the algorithm selects a vector from a decision set after which it receives a noisy linear loss parameterized by an unknown vector. The goal in such a problem is to minimize the (pseudo) regret which is the difference between the total expected …

New model-free algorithm achieves similar LQR regret guarantees.

We consider stochastic multi-armed bandit problems with graph feedback, where the decision maker is allowed to observe the neighboring actions of the chosen action. We allow the graph structure to vary with time and consider both deterministic and Erdős-Rényi random graph models. For such a graph feedback model, we fir…

Online learning to rank is a core problem in machine learning. In Lattimore et al. (2018), a novel online learning algorithm was proposed based on topological sorting. In the paper they provided a set of self-normalized inequalities (a) in the algorithm as a criterion in iterations and (b) to provide an upper bound for…

The paper introduces a method to learn and apply value envelopes for faster online reinforcement learning.

We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By …