Paper improves worst-case regret bounds for RLSVI in reinforcement learning.
problem Minimizing regret in reinforcement learning with randomized value functions.
method Introduces a clipping variant of Thompson Sampling for RLSVI.
result Achieves a i l d e O ( H 2 S A T ) ilde{\mathrm{O}}(H^2S\sqrt{AT}) i l d e O ( H 2 S A T ) worst-case regret bound. New algorithm reduces worst-case regret for heavy-tailed bandits.
problem Stochastic Multi-Armed Bandit problem with heavy-tailed rewards.
method Modified minimax policy MOSS with saturated empirical mean.
result Worst-case regret matching lower bound for heavy-tailed distributions.
This paper analyzes how randomizing rewards in MBRL can improve performance without being overly optimistic.
problem The gap between theoretical worst-case regret analysis and empirical performance in MBRL.
method Reward randomization in model-based reinforcement learning (MBRL) with kernelized linear regulator (KNR) model.
result Reward randomization guarantees partial optimism and near-optimal worst-case regret.
New algorithm minimizes worst-case regret in uncertain, time-varying dynamics.
problem Model-based policy learning in uncertain, time-varying dynamics.
method Planning regret metric and iterative algorithm for minimizing it.
result Empirical evidence shows the proposed algorithm outperforms existing methods.
New policy optimizes risk and optimality in stochastic bandits.
problem Optimizing risk in stochastic bandits with heavy-tailed risk.
method Designing policies with worst-case optimality for expected regret and light-tailed risk distribution.
result Achieves worst-case optimality for expected regret and light-tailed risk distribution.
A new algorithm avoids worst-case outcomes in risky contexts.
problem Risk-averse behavior in contextual bandits is challenging.
method Developed a first risk-averse contextual bandit algorithm with online regret guarantees.
result First algorithm with an online regret guarantee for risk-averse contextual bandits.
Bandits with Knapsacks (BwK) is a general model for multi-armed bandits under supply/budget constraints. While worst-case regret bounds for BwK are well-understood, we present three results that go beyond the worst-case perspective. First, we provide upper and lower bounds which amount to a full characterization for lo…
New metric measures variability in bandit algorithms, linking regret and variability.
problem Variability in multi-armed bandit allocations harms modern applications.
method Introduces allocation variability as a new metric and establishes a trade-off with regret.
result Any minimax regret-optimal algorithm must incur worst-case allocation variability Θ(T).
This paper studies a recent proposal to use randomized value functions to drive exploration in reinforcement learning. These randomized value functions are generated by injecting random noise into the training data, making the approach compatible with many popular methods for estimating parameterized value functions. B…
New approach limits regret in non-stationary bandits.
problem Understanding worst case regret in time-varying bandits.
method Belief inertia argument to resist new evidence after changes.
result Linear worst case regret for classical and restarting algorithms.
Optimal strategy identified for minimizing regret in fixed-budget best arm selection.
problem Minimizing expected simple regret in fixed-budget best arm selection.
method Two-Stage (TS)-Hirano-Imbens-Ridder (HIR) strategy using HIR estimator.
result TS-HIR strategy is asymptotically minimax optimal.
Improved algorithms for stochastic linear bandits using tighter confidence sequences.
problem Stochastic linear bandits with improved worst-case regret guarantees.
method Novel tail bound for adaptive martingale mixtures to construct tighter confidence sequences.
result Linear bandit algorithm achieves competitive worst-case regret.
Quantum RL algorithm achieves logarithmic regret for exploration.
problem Designing efficient quantum RL algorithms for exploration.
method UCRL-style quantum algorithm with lazy updating and quantum estimation.
result Proves O ( p o l y ( S , A , H , log T ) ) \mathcal{O}(\mathrm{poly}(S, A, H, \log T)) O ( poly ( S , A , H , log T )) worst-case regret. Improved online classification with accurate predictions.
problem Online classification challenges with limited data.
method Designing an online learner that uses predictions to reduce regret.
result Expected regret is better than worst-case analysis, especially with accurate predictions.
Follow-the-Leader (FTL) is an intuitive sequential prediction strategy that guarantees constant regret in the stochastic setting, but has terrible performance for worst-case data. Other hedging strategies have better worst-case guarantees but may perform much worse than FTL if the data are not maximally adversarial. We…
New algorithm reduces regret for kernelized bandits by adapting to specific problem instances.
problem Efficiently learning the optimizer of an unknown function in RKHS with noisy oracle.
method Instance-dependent regret analysis and a new minimax near-optimal algorithm.
result New algorithm achieves better performance on specific problem instances.
Optimizes regret distribution in stochastic bandits for risk balance.
problem Balancing regret expectation and tail risk in stochastic bandits.
method Characterizes optimal regret tail probability for any threshold, proposes new policies.
result Discovers an intrinsic gap in optimal tail rate based on time horizon uncertainty.
A scheme robust to action erasures improves MAB performance.
problem Learning in multi-arm bandits with erasure channels.
method Proposes a scheme to work on top of any MAB algorithm.
result Worst-case regret is at most a factor of O ( 1 / 1 − ε ) O(1/\sqrt{1-ε}) O ( 1/ 1 − ε ) away from no-erasure case. Improves policy optimization with polylog(T) regret bounds for stochastic losses.
problem Improves theoretical guarantees for policy optimization in stochastic settings.
method Leverages Tsallis and Shannon entropy regularizers for polylog(T) regret, and log-barrier regularizer for adversarial settings.
result Achieves a first-order polylog(T) regret bound for policy optimization in stochastic settings.
This paper calculates worst-case target semi-variances for uncertain losses.
problem Managing risk when loss distribution is uncertain and only partial information is known.
method Derives worst-case target semi-variances for symmetric or non-negative losses under uncertainty sets representing investor's undesirable scenarios.
result Closed-form expressions for worst-case target semi-variances are derived.
New algorithms achieve better regret bounds for online classification with relaxed benchmarks.
problem Competing with worst-case optimal binary loss in online classification.
method Comparing against predictors robust to small input perturbations, performing well under Gaussian smoothing, or maintaining a prescribed output margin.
result Regret guarantees depend only on VC dimension and instance space complexity, with an O ( log ( 1 / γ ) ) O(\log(1/γ)) O ( log ( 1/ γ )) dependence on the generalized margin. New reinforcement learning algorithm achieves instance-optimal sample complexity.
problem Achieving low regret and identifying optimal policies in reinforcement learning.
method A novel planning-based algorithm that explicitly accounts for state visitation distributions.
result The proposed algorithm attains nearly minimax optimal sample complexity, improving over worst-case bounds.
We investigate online convex optimization in changing environments, and choose the adaptive regret as the performance measure. The goal is to achieve a small regret over every interval so that the comparator is allowed to change over time. Different from previous works that only utilize the convexity condition, this pa…
Optimal algorithm for identifying best-arm with minimal regret.
problem Identifying the best arm in two treatments with limited budget.
method Neyman allocation based on outcome standard deviations.
result Neyman allocation is minimax optimal for simple regret.
Optimized algorithms for online learning with linear constraints improve performance and provide worst-case analysis.
problem Improving online learning algorithms for constrained optimization problems.
method Developed an optimized variant of an online Frank-Wolfe algorithm and used semidefinite programming for numerical analysis.
result No pure online Frank-Wolfe algorithm can have a better regret guarantee than O(T^3/4) without additional assumptions.
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
UCB algorithm's arm-sampling behavior is revealed, leading to new insights and proofs.
problem Optimizing multi-armed bandit algorithms for worst-case scenarios.
method Analysis of UCB algorithm's arm-sampling behavior and process-level characterization.
result UCB's arm-sampling rates are asymptotically deterministic, regardless of problem complexity.
Oracle-efficient algorithms for online learning with smoothed and hint-adversaries.
problem Online learning with beyond worst-case adversaries.
method Oracle-efficient algorithms for two settings: smoothed analysis and K K K -hint transductive learning. result Oracle-efficient regret bounds for learning real-valued and binary-valued functions.
We consider online learning problems where the aim is to achieve regret which is efficient in the sense that it is the same order as the lowest regret amongst K experts. This is a substantially stronger requirement that achieving O ( n ) O(\sqrt{n}) O ( n ) or O ( log n ) O(\log n) O ( log n ) regret with respect to the best expert and standard algorithm…
New betting strategy reduces regret to ln(ln n) with protection against adversarial data.
problem Tackles the problem of minimizing regret in betting against adversarial and stochastic data.
method Combines insights from Robbins and Cover, using a mixture strategy.
result Exhibits a regret of O(ln(ln n)) on almost all paths, with O(log n) regret on the complement.
New algorithm expands FTRL framework with improved worst-case regret bounds.
problem Online learning with improved worst-case regret bounds.
method Generalized implicit Follow-The-Regularized-Leader (FTRL) algorithm.
result Unified framework for designing updates improving worst-case regret bounds.
Most methods for decision-theoretic online learning are based on the Hedge algorithm, which takes a parameter called the learning rate. In most previous analyses the learning rate was carefully tuned to obtain optimal worst-case performance, leading to suboptimal performance on easy instances, for example when there ex…
We present methods for online linear optimization that take advantage of benign (as opposed to worst-case) sequences. Specifically if the sequence encountered by the learner is described well by a known "predictable process", the algorithms presented enjoy tighter bounds as compared to the typical worst case bounds. Ad…
New algorithms for neural bandits learn from context and arm features.
problem Learning from context and arm features in combinatorial bandit problems.
method Proposed algorithms CN-UCB and CN-TS using deep neural networks.
result Achieved regret guarantees for the first time in combinatorial neural bandits.
Algorithm allocates budgets to tasks with semi-bandit feedback, achieving near-optimal regret bounds.
problem Stochastic budget allocation with censored semi-bandit feedback.
method Optimism-based algorithm operating under censored semi-bandit feedback.
result Regret scales polylogarithmically with horizon T in diminishing-returns regimes.
The paper tackles fair sequential decision making with biased linear bandit feedback.
problem Fair sequential decision making with biased linear bandit feedback.
method Phased elimination algorithm to correct unfair evaluations, establishing upper bounds on regret.
result The worst-case regret is smaller than O ( κ ∗ 1 / 3 log ( T ) 1 / 3 T 2 / 3 ) \mathcal{O}(κ_*^{1/3}\log(T)^{1/3}T^{2/3}) O ( κ ∗ 1/3 log ( T ) 1/3 T 2/3 ) . Algorithm tackles adaptive discretization in adversarial Lipschitz bandits for dynamic pricing and auctions.
problem Adaptive discretization in adversarial Lipschitz bandits.
method Adversarial Zooming algorithm for adaptive discretization.
result First algorithm for adversarial Lipschitz bandits with instance-dependent regret bounds.
Paper improves RL algorithms with known optimal Q-value predictions.
problem Improving RL algorithms with known optimal Q-value predictions.
method Shows how to improve regret bounds by distilling known optimal Q-value predictions.
result Achieves sublinear regret with known predictions, even if not distillation.
Paper improves sparse linear bandits by accounting for noise variance.
problem Sparse linear bandits with unknown noise variance.
method Develops a general framework to convert variance-aware algorithms to sparse linear bandits.
result Achieves $\widetilde{\mathcal O}\left(\sqrt{d\sum_{t=1}^T σ_t^2} + 1
ight)$ regret, interpolating between worst-case and benign settings.
We prove a new minimax theorem connecting the worst-case Bayesian regret and minimax regret under partial monitoring with no assumptions on the space of signals or decisions of the adversary. We then generalise the information-theoretic tools of Russo and Van Roy (2016) for proving Bayesian regret bounds and combine th…
We introduce a modular framework for market making. It combines cost-function based automated market makers with bandit algorithms. We obtain worst-case profits guarantee's relative to the best in hindsight within a class of natural "overround" cost functions . This combination allow us to have distribution-free guaran…
Study online learning in MDPs with aggregate bandit feedback, achieving low regret in both stochastic and adversarial settings.
problem Online learning in finite-horizon episodic MDPs with aggregate bandit feedback.
method Best-of-both-worlds (BOBW) algorithms using FTRL over occupancy measures, self-bounding techniques, and new loss estimators.
result First BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback achieving O ( log T ) O(\log T) O ( log T ) regret in stochastic and O ( T ) {O}(\sqrt{T}) O ( T ) regret in adversarial settings. The paper tackles robust policy learning from multiple data sources.
problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.
method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.
result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.
This paper improves Q-learning bounds using reference-advantage decomposition.
problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.
method Develops a novel error decomposition framework to prove gap-dependent regret bounds.
result Establishes logarithmic gap-dependent regret bounds for Q-learning.
Improved regret bounds for adversarial linear contextual bandits.
problem Adversarial linear contextual bandits with changing loss functions.
method Truncated continuous exponential weights algorithm over the probability simplex, analyzing with linear bandit setting without contexts.
result Second-order bound of i l d e O ( K d V T ) ilde O(K\sqrt{d V_T}) i l d e O ( K d V T ) and first-order bound of i l d e O ( K d L T ∗ ) ilde O(K\sqrt{d L_T^*}) i l d e O ( K d L T ∗ ) . New algorithm reduces reinforcement learning regret by adapting to interaction variability.
problem Existing reinforcement learning methods lack adaptability to interaction variability.
method Developed a variance-adaptive optimal algorithm for MNL function approximation.
result Achieved instance-wise optimal regret bounds, validating efficiency in practice.
New bounds for online portfolio selection without smoothness assumptions.
problem Online portfolio selection with non-Lipschitz, non-smooth losses.
method Data-dependent bounds using novel smoothness characterizations and FTRL with self-concordant regularizers.
result Achieves logarithmic regrets when data is 'easy' and sublinear worst-case regrets.
Improved FTPL algorithm reduces regret in predictable minimax games.
problem Online learning and minimax games with predictable loss sequences.
method Optimistic modification of FTPL with dual regularization view.
result Tighter regret bounds for predictable sequences, O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 ) accuracy.