arXiv research

Improved gap-dependent bounds for reinforcement learning with linear approximations.

This paper improves Q-learning bounds using reference-advantage decomposition.

Study gap-dependent regret bounds for risk-sensitive RL.

New method reduces regret and communication costs in federated Q-learning.

Fine-grained gap-dependent regret bounds for reinforcement learning.

This paper establishes that optimistic algorithms attain gap-dependent and non-asymptotic logarithmic regret for episodic MDPs. In contrast to prior work, our bounds do not suffer a dependence on diameter-like quantities or ergodicity, and smoothly interpolate between the gap dependent logarithmic-regret, and the $\wid…

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

Unified framework for corruption-robust linear bandits with optimal gap-dependent misspecification bounds.

New algorithm reduces MDP regret by accounting for state suboptimality gaps and variance.

A matroid is a notion of independence in combinatorial optimization which is closely related to computational efficiency. In particular, it is well known that the maximum of a constrained modular function can be found greedily if and only if the constraints are associated with a matroid. In this paper, we bring togethe…

New algorithm reduces regret in private online learning with optimal gap-dependent rate.

Adaptive smooth non-stationary bandits achieve optimal regret rates without knowing parameters.

We consider the noisy power method algorithm, which has wide applications in machine learning and statistics, especially those related to principal component analysis (PCA) under resource (communication, memory or privacy) constraints. Existing analysis of the noisy power method shows an unsatisfactory dependency over …

Algorithm reduces regret in misspecified linear contextual bandits.

New algorithms minimize regret in streaming MAB with memory constraints.

We make three contributions to the theory of k-armed adversarial bandits. First, we prove a first-order bound for a modified variant of the INF strategy by Audibert and Bubeck [2009], without sacrificing worst case optimality or modifying the loss estimators. Second, we provide a variance analysis for algorithms based …

Many important optimization problems, such as the minimum spanning tree and minimum-cost flow, can be solved optimally by a greedy method. In this work, we study a learning variant of these problems, where the model of the problem is unknown and has to be learned by interacting repeatedly with the environment in the ba…

New RL algorithms reduce costs for single-agent and federated learning.

A search engine usually outputs a list of $K$ web pages. The user examines this list, from the first web page to the last, and chooses the first attractive page. This model of user behavior is known as the cascade model. In this paper, we propose cascading bandits, a learning variant of the cascade model where the obje…

A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to constraints, and then observes stochastic weights of these items and receives their sum as a payoff. In this paper, we close the problem of computationally and sample effi…

Improves policy optimization with polylog(T) regret bounds for stochastic losses.

New algorithms identify best policies in discounted linear MDPs efficiently.

KL-MS improves regret bounds for multi-armed bandits with bounded rewards.

RAVEN-UCB addresses non-stationary MAB problems with tighter regret bounds.

Non-stationarity appears in many online applications such as web search and advertising. In this paper, we study the online learning to rank problem in a non-stationary environment where user preferences change abruptly at an unknown moment in time. We consider the problem of identifying the K most attractive items and…

We propose an online algorithm for cumulative regret minimization in a stochastic multi-armed bandit. The algorithm adds $O(t)$ i.i.d. pseudo-rewards to its history in round $t$ and then pulls the arm with the highest average reward in its perturbed history. Therefore, we call it perturbed-history exploration (PHE). Th…

New algorithm reduces sample complexity for planning in MDPs.

PAC-Bayes bound for stable RNNs in time-series data.

In deep neural networks, the spectral norm of the Jacobian of a layer bounds the factor by which the norm of a signal changes during forward/backward propagation. Spectral norm regularizations have been shown to improve generalization, robustness and optimization of deep learning methods. Existing methods to compute th…

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

Unified framework for distributional regret in bandits and reinforcement learning.

A search engine recommends to the user a list of web pages. The user examines this list, from the first page to the last, and clicks on all attractive pages until the user is satisfied. This behavior of the user can be described by the dependent click model (DCM). We propose DCM bandits, an online learning variant of t…

Paper develops robust estimators and strategies for stochastic MABs with heavy-tailed rewards.

Improved reinforcement learning for episodes with varying action sets.

In this paper, we study the combinatorial multi-armed bandit problem (CMAB) with probabilistically triggered arms (PTAs). Under the assumption that the arm triggering probabilities (ATPs) are positive for all arms, we prove that a class of upper confidence bound (UCB) policies, named Combinatorial UCB with exploration …

Online learning to rank is a core problem in information retrieval and machine learning. Many provably efficient algorithms have been recently proposed for this problem in specific click models. The click model is a model of how the user interacts with a list of documents. Though these results are significant, their im…

New algorithms reduce regret in online MDPs by adapting to data and variance.

The paper tackles resource allocation for arms with unknown and random rewards, achieving optimal regret bounds.

New insights into balancing reward and fairness in stochastic MAB.

We consider the dynamic assortment optimization problem under the multinomial logit model (MNL) with unknown utility parameters. The main question investigated in this paper is model mis-specification under the $\varepsilon$-contamination model, which is a fundamental model in robust statistics and machine learning. In…

Motivated by applications in computational advertising and systems biology, we consider the problem of identifying the best out of several possible soft interventions at a source node $V$ in an acyclic causal directed graph, to maximize the expected value of a target node $Y$ (located downstream of $V$). Our setting im…

The paper tackles fair sequential decision making with biased linear bandit feedback.

New algorithm optimizes multi-armed bandit performance in stochastic and adversarial settings.

Logarithmic regret achieved in RL with linear function approximation.

Algorithm optimizes multi-player learning with noisy rewards without direct communication.

Local gaps in Ricci shrinkers depend only on dimension.

We consider the setup of stochastic multi-armed bandits in the case when reward distributions are piecewise i.i.d. and bounded with unknown changepoints. We focus on the case when changes happen simultaneously on all arms, and in stark contrast with the existing literature, we target gap-dependent (as opposed to only g…

Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.