arXiv research

This paper improves Q-learning bounds using reference-advantage decomposition.

New method reduces regret and communication costs in federated Q-learning.

Improved gap-dependent bounds for reinforcement learning with linear approximations.

Fine-grained gap-dependent regret bounds for reinforcement learning.

This paper establishes that optimistic algorithms attain gap-dependent and non-asymptotic logarithmic regret for episodic MDPs. In contrast to prior work, our bounds do not suffer a dependence on diameter-like quantities or ergodicity, and smoothly interpolate between the gap dependent logarithmic-regret, and the $\wid…

Study gap-dependent regret bounds for risk-sensitive RL.

Unified framework for corruption-robust linear bandits with optimal gap-dependent misspecification bounds.

We consider the noisy power method algorithm, which has wide applications in machine learning and statistics, especially those related to principal component analysis (PCA) under resource (communication, memory or privacy) constraints. Existing analysis of the noisy power method shows an unsatisfactory dependency over …

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

New algorithm reduces MDP regret by accounting for state suboptimality gaps and variance.

New algorithm reduces regret in private online learning with optimal gap-dependent rate.

New insights into balancing reward and fairness in stochastic MAB.

Adaptive smooth non-stationary bandits achieve optimal regret rates without knowing parameters.

A matroid is a notion of independence in combinatorial optimization which is closely related to computational efficiency. In particular, it is well known that the maximum of a constrained modular function can be found greedily if and only if the constraints are associated with a matroid. In this paper, we bring togethe…

We propose an online algorithm for cumulative regret minimization in a stochastic multi-armed bandit. The algorithm adds $O(t)$ i.i.d. pseudo-rewards to its history in round $t$ and then pulls the arm with the highest average reward in its perturbed history. Therefore, we call it perturbed-history exploration (PHE). Th…

We make three contributions to the theory of k-armed adversarial bandits. First, we prove a first-order bound for a modified variant of the INF strategy by Audibert and Bubeck [2009], without sacrificing worst case optimality or modifying the loss estimators. Second, we provide a variance analysis for algorithms based …

New RL algorithms reduce costs for single-agent and federated learning.

Paper develops robust estimators and strategies for stochastic MABs with heavy-tailed rewards.

New algorithms minimize regret in streaming MAB with memory constraints.

Algorithm reduces regret in misspecified linear contextual bandits.

Local gaps in Ricci shrinkers depend only on dimension.

Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.

Graph Laplacians computed from weighted adjacency matrices are widely used to identify geometric structure in data, and clusters in particular; their spectral properties play a central role in a number of unsupervised and semi-supervised learning algorithms. When suitably scaled, graph Laplacians approach limiting cont…

Logarithmic regret strategies for safe multi-armed bandits with safety risk constraints.

Many important optimization problems, such as the minimum spanning tree and minimum-cost flow, can be solved optimally by a greedy method. In this work, we study a learning variant of these problems, where the model of the problem is unknown and has to be learned by interacting repeatedly with the environment in the ba…

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

New algorithm reduces sample complexity for planning in MDPs.

The paper tackles fair sequential decision making with biased linear bandit feedback.

A search engine usually outputs a list of $K$ web pages. The user examines this list, from the first web page to the last, and chooses the first attractive page. This model of user behavior is known as the cascade model. In this paper, we propose cascading bandits, a learning variant of the cascade model where the obje…

Online learning to rank is a core problem in information retrieval and machine learning. Many provably efficient algorithms have been recently proposed for this problem in specific click models. The click model is a model of how the user interacts with a list of documents. Though these results are significant, their im…

A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to constraints, and then observes stochastic weights of these items and receives their sum as a payoff. In this paper, we close the problem of computationally and sample effi…

Motivated by applications in computational advertising and systems biology, we consider the problem of identifying the best out of several possible soft interventions at a source node $V$ in an acyclic causal directed graph, to maximize the expected value of a target node $Y$ (located downstream of $V$). Our setting im…

Non-stationarity appears in many online applications such as web search and advertising. In this paper, we study the online learning to rank problem in a non-stationary environment where user preferences change abruptly at an unknown moment in time. We consider the problem of identifying the K most attractive items and…

Flat minimal tori counterexamples refute Lu's second-gap conjecture.

RAVEN-UCB addresses non-stationary MAB problems with tighter regret bounds.

A search engine recommends to the user a list of web pages. The user examines this list, from the first page to the last, and clicks on all attractive pages until the user is satisfied. This behavior of the user can be described by the dependent click model (DCM). We propose DCM bandits, an online learning variant of t…

Improves policy optimization with polylog(T) regret bounds for stochastic losses.

New method for identifying best designs in vector optimization with uncertain feedback.

We propose combinatorial cascading bandits, a class of partial monitoring problems where at each step a learning agent chooses a tuple of ground items subject to constraints and receives a reward if and only if the weights of all chosen items are one. The weights of the items are binary, stochastic, and drawn independe…

New algorithms identify best policies in discounted linear MDPs efficiently.

KL-MS improves regret bounds for multi-armed bandits with bounded rewards.

Linearized attention fails to converge to NTK limit even at large widths.

ReLEX algorithm improves RL efficiency by selecting optimal representations.

Algorithm optimizes multi-player learning with noisy rewards without direct communication.

In this paper, we study the combinatorial multi-armed bandit problem (CMAB) with probabilistically triggered arms (PTAs). Under the assumption that the arm triggering probabilities (ATPs) are positive for all arms, we prove that a class of upper confidence bound (UCB) policies, named Combinatorial UCB with exploration …

Unified framework for distributional regret in bandits and reinforcement learning.

The paper tackles resource allocation for arms with unknown and random rewards, achieving optimal regret bounds.

Logarithmic regret achieved in RL with linear function approximation.