Paper tackles online learning with interval regret, achieving adaptive bounds.
problem Non-stationary online learning over time intervals.
method Two-layer online ensemble structure with gradient variation.
result Achieves strong theoretical guarantees with adaptive bounds.
Regret minimization is treated as the golden rule in the traditional study of online learning. However, regret minimization algorithms tend to converge to the static optimum, thus being suboptimal for changing environments. To address this limitation, new performance measures, including dynamic regret and adaptive regr…
Efficient methods reduce projections in non-stationary online learning.
problem Optimizing dynamic and adaptive regret in non-stationary online learning environments.
method Presented efficient methods reducing the number of projections per round from O ( log T ) O(\log T) O ( log T ) to 1 1 1 . result Reduced number of projections per round from O ( log T ) O(\log T) O ( log T ) to 1 1 1 for optimizing dynamic and adaptive regret. New algorithm reduces control error in systems with changing dynamics.
problem Online control of systems with time-varying linear dynamics.
method Introduces adaptive regret metric and a novel meta-algorithm.
result First adaptive regret bound for online convex optimization with memory.
Optimal online linear regression in dynamic environments using discounted Vovk-Azoury-Warmuth forecaster.
problem Achieving optimal performance in dynamic online linear regression without prior knowledge.
method Developed a discounted variant of the Vovk-Azoury-Warmuth forecaster to achieve optimal dynamic regret guarantees.
result Achieved dynamic regret of the form $O\left(d\log(T)\vee \sqrt{dP_{T}^γ(\vec{u})T}
ight)$ , with a learnable discount factor.
New online conformal prediction methods minimize strongly adaptive regret and achieve near-optimal coverage.
problem Uncertainty quantification in online settings with changing data distributions.
method Developed new online conformal prediction methods that minimize strongly adaptive regret.
result Achieve near-optimal strongly adaptive regret and approximately valid coverage.
Near-logarithmic regret per switch achieved for mixable/exp-concave losses.
problem Online optimization of mixable loss functions with dynamic environments.
method Online mixture framework using static solvers and hyper-expert creations.
result Near-logarithmic regret per switch with sub-polynomial complexity.
Research aims to improve confidence intervals for RKHS elements in online learning.
problem Improper confidence intervals lead to suboptimal regret bounds in kernel-based bandit and reinforcement learning.
method Formalizes the open problem of online confidence intervals in RKHS and reviews existing results.
result Identifies the online nature of observation points as the main challenge for tight confidence intervals.
Consider the classical problem of predicting the next bit in a sequence of bits. A standard performance measure is {\em regret} (loss in payoff) with respect to a set of experts. For example if we measure performance with respect to two constant experts one that always predicts 0's and another that always predicts 1's …
Most contextual bandit algorithms minimize regret against the best fixed policy, a questionable benchmark for non-stationary environments that are ubiquitous in applications. In this work, we develop several efficient contextual bandit algorithms for non-stationary environments by equipping existing methods for i.i.d. …
Algorithm tackles adaptive discretization in adversarial Lipschitz bandits for dynamic pricing and auctions.
problem Adaptive discretization in adversarial Lipschitz bandits.
method Adversarial Zooming algorithm for adaptive discretization.
result First algorithm for adversarial Lipschitz bandits with instance-dependent regret bounds.
We investigate online convex optimization in changing environments, and choose the adaptive regret as the performance measure. The goal is to achieve a small regret over every interval so that the comparator is allowed to change over time. Different from previous works that only utilize the convexity condition, this pa…
LqgOpt learns optimal control in unknown LQG systems with minimal regret.
problem Adaptive control in partially observable linear quadratic Gaussian systems with unknown dynamics.
method Optimism in the face of uncertainty, predictor state evolution, closed-loop system identification, confidence bounds.
result Proves a regret upper bound of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) for LQG systems. We consider online algorithms under both the competitive ratio criteria and the regret minimization one. Our main goal is to build a unified methodology that would be able to guarantee both criteria simultaneously. For a general class of online algorithms, namely any Metrical Task System (MTS), we show that one can sim…
Optimal algorithms for mixable losses in dynamic environments with reduced redundancy.
problem Online optimization of mixable loss functions in a dynamic environment.
method Introduce online mixture schemes with polynomial and logarithmic time complexities.
result Achieves optimal redundancy up to a constant multiplicity gap.
Optimal simple regret bound for Gaussian Process bandits.
problem Sequential optimization of expensive-to-evaluate functions.
method Proved a bound on simple regret for pure exploration algorithms.
result Order optimal bound on simple regret for Gaussian Process bandits.
The paper tackles lifelong learning in multi-armed bandits, aiming to minimize average regret over multiple tasks.
problem Minimizing average regret in multi-armed bandits over multiple tasks.
method Confidence interval tuning of UCB algorithms and greedy algorithms applied to a bandit over bandit approach.
result Empirical improvement over previous work in the mortal bandit problem.
New algorithm reduces adaptive regret without projections.
problem Computational expense of projections in online convex optimization.
method Lazy gradient-based algorithm with set-membership computations.
result Near-optimal adaptive regret bounds for general convex functions.
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
Kernel-based function approximation improves reinforcement learning performance.
problem Average reward reinforcement learning in infinite horizon settings.
method Optimistic algorithm based on kernel ridge regression.
result No-regret performance guarantees and confidence intervals for kernel-based predictions.
Efficiently solves exploration-exploitation in LQR using Lagrangian relaxation.
problem Exploration-exploitation dilemma in linear quadratic regulator (LQR) setting.
method Relax optimistic optimization into a constrained extended LQR problem, then solve using Riccati equations.
result Computes ε ε ε -optimistic controller efficiently with O ( log ( 1 / ε ) ) O\big(\log(1/ε)\big) O ( log ( 1/ ε ) ) Riccati equations. A new UCB policy improves reward-cost ratio estimation in budgeted MAB.
problem Maximizing reward-cost ratio under budget constraints.
method ω-UCB policy with asymmetric confidence intervals.
result Logarithmic regret and superior performance in various settings.
New algorithms reduce dynamic regret in non-stationary RL environments.
problem Optimizing policies in environments that change over time.
method POWER and POWER++ algorithms for policy optimization with dynamic regret analysis.
result POWER++ improves dynamic regret by actively adapting to non-stationarity.
New algorithms achieve optimal regret in sliding window model with limited memory.
problem Experts problem in the sliding window model with limited information.
method 2 queries, polylog(nT) memory, exponential improvement on memory.
result Achieve optimal regret of sqrt(nW)polylog(nT) with 2 queries and polylog(nT) memory.
Two algorithms for linear contextual bandits with rare updates achieve optimal regret and efficiency.
problem Linear contextual bandits with infrequent parameter updates.
method Two practical algorithms with O ( log log T ) O(\log\log T) O ( log log T ) updates, BLCE-G and BLCE. result Minimax-optimal regret with low computational complexity.
Most bandit algorithm designs are purely theoretical. Therefore, they have strong regret guarantees, but also are often too conservative in practice. In this work, we pioneer the idea of algorithm design by minimizing the empirical Bayes regret, the average regret over problem instances sampled from a known distributio…
In online learning, the dynamic regret metric chooses the reference (optimal) solution that may change over time, while the typical (static) regret metric assumes the reference solution to be constant over the whole time horizon. The dynamic regret metric is particularly interesting for applications such as online reco…
Paper develops a discounted algorithm for online convex optimization that adapts to unknown discount factors.
problem Developing an algorithm that can adapt to an unknown discount factor in online convex optimization.
method Smoothed Online Gradient Descent (SOGD) with Discounted-Normal-Predictor (DNP).
result Achieves a uniform O ( log T / 1 − λ ) O(\sqrt{\log T/1-λ}) O ( log T /1 − λ ) discounted regret across a continuous interval of discount factors. Improved online algorithm for convex losses with near-optimal swap regret.
problem Minimizing swap regret for convex losses.
method Randomized online algorithm using multi-scale binning.
result Near-optimal O ~ ( T ) \widetilde O(\sqrt T) O ( T ) expected swap regret. Optimistic Hedge achieves optimal regret bounds in two-player zero-sum games.
problem Achieving optimal regret bounds for optimistic Hedge in two-player zero-sum games.
method Refined regret analysis and optimization problem formulation.
result Optimistic Hedge achieves O ( log m log n ) O(\sqrt{\log m \log n}) O ( log m log n ) regret bounds, matching upper and lower bounds. Recursive least-squares algorithms often use forgetting factors as a heuristic to adapt to non-stationary data streams. The first contribution of this paper rigorously characterizes the effect of forgetting factors for a class of online Newton algorithms. For exp-concave and strongly convex objectives, the algorithms a…
Reduces dynamic regret to static problem in RKHS.
problem Minimizing cumulative loss in online convex optimization.
method Reduces dynamic regret to static regret problem in RKHS.
result Optimal dynamic regret guarantees for linear losses and new bounds for exp-concave and improper linear regression.
SCaLE tackles dynamic regret in noisy bandit feedback with switching costs.
problem Unbounded metric movement costs in bandit online convex optimization.
method SCaLE algorithm for high-dimensional dynamic quadratic hitting costs and ℓ 2 \ell_2 ℓ 2 -norm switching costs, with spectral regret analysis. result First algorithm achieving sub-linear dynamic regret without hitting cost knowledge.
Online learning is a powerful tool for analyzing iterative algorithms. However, the classic adversarial setup sometimes fails to capture certain regularity in online problems in practice. Motivated by this, we establish a new setup, called Continuous Online Learning (COL), where the gradient of online loss function cha…
Improved algorithm for adaptive dueling bandits with near-optimal regret bound.
problem Non-stationary dueling bandits with unknown number of preference changes.
method Elimination-based rescheduling algorithm for adaptive dynamic regret.
result Near-optimal i l d e O ( S e x t t t C W T ) ilde{O}(\sqrt{S^{ exttt{CW}} T}) i l d e O ( S e x ttt C W T ) dynamic regret bound. New algorithms balance collaboration and adversarial behavior in linear bandits.
problem Minimizing regret in a collaborative linear bandit problem with adversarial agents.
method Robust collaborative phased elimination algorithm with tight analyses.
result Achieves near-optimal regret bounds of $O\left(α+ 1/\sqrt{M}
ight) \sqrt{dT}$ for good agents.
Develops a new method for online conformal prediction without manual tuning.
problem Achieving long-run 1 − α 1-α 1 − α coverage for arbitrary data streams in an informative manner. method Linearized regret theory and universal portfolio algorithms.
result Strong finite-time bounds on miscoverage for UP-OCP, outperforming prior methods.
As a metric to measure the performance of an online method, dynamic regret with switching cost has drawn much attention for online decision making problems. Although the sublinear regret has been provided in many previous researches, we still have little knowledge about the relation between the dynamic regret and the s…
New algorithm reduces dynamic regret without prior function change knowledge.
problem Non-stationary stochastic optimization with bandit feedback.
method Fixed step sizes combined with multi-scale sampling framework.
result Achieves optimal dynamic regret without prior function change knowledge.
New algorithm reduces dynamic regret in time-varying movement costs.
problem Dynamic regret in online convex optimization with time-varying movement costs.
method Introduced a novel algorithm for time-varying movement costs, achieving comparator-adaptive dynamic regret bound.
result Established first comparator-adaptive dynamic regret bound of O ~ ( ( M 2 + M P T ) ( T + ∑ t λ t ) ) \widetilde{\mathcal{O}}(\sqrt{(M^2+MP_T)(T+\sum_t λ_t)}) O ( ( M 2 + M P T ) ( T + ∑ t λ t ) ) . New algorithm achieves both static and dynamic regret optimally against an oblivious adversary for deterministic losses.
problem Achieving optimal static and dynamic regret simultaneously in adversarial bandits.
method Extends impossibility result to deterministic losses, uses negative static regret and Blackwell approachability.
result First algorithm achieving optimal static and dynamic regret simultaneously against an oblivious adversary.
New algorithms reduce dynamic regret in online MDPs with changing losses.
problem Online MDPs with adversarial loss changes and known transitions.
method Dynamic regret measure, novel ensemble algorithms for three models.
result Provably optimal dynamic regret bounds for episodic SSP, improved bounds for predictable environments.
Sequential screening and dynamic regret in multi-armed bandits with arriving arms
problem Sequential experimentation with expanding arm set
method UCB-AA with preliminary screening
result Regret bounds depend on arrival process
New algorithms minimize dynamic regret for strongly convex losses.
problem Minimizing dynamic regret for strongly convex losses.
method Developed Strongly Adaptive algorithms exploiting KKT conditions.
result Achieved near optimal dynamic regret of O ( d 1 / 3 n 1 / 3 e x t T V [ u 1 : n ] 2 / 3 ∨ d ) O(d^{1/3} n^{1/3} ext{TV}[u_{1:n}]^{2/3} \vee d) O ( d 1/3 n 1/3 e x t T V [ u 1 : n ] 2/3 ∨ d ) . SA-BCP combines long-term and local evidence for efficient, adaptive online prediction.
problem Balancing fast adaptation and stable coverage in online prediction.
method State-Adaptive Bayesian Conformal Prediction (SA-BCP) using gated convex combination of temporal inertia and spatial evidence.
result SA-BCP achieves at-or-above-nominal coverage with substantially sharper intervals compared to discounted Bayesian CP.
Improved dynamic regret analysis for strongly convex and smooth functions.
problem Analyzing dynamic regret for online learning algorithms.
method Improved analysis of the Online Multiple Gradient Descent (OMGD) algorithm.
result Achieved a best-of-three-worlds guarantee for dynamic regret.
Variational autoencoder models dynamic latent graphs for neural point processes.
problem Modeling event dynamics with changing trends over time.
method Sequential latent variable model with dynamic latent graphs.
result Higher accuracy in predicting inter-event times and event types.
Dynamic regret minimization is shown equivalent to static regret minimization for linear losses.
problem Dynamic regret minimization in online convex optimization.
method Equivalence between dynamic and static regret minimization for linear losses.
result Dynamic regret minimization is equivalent to static regret minimization for linear losses.