Efficient algorithms for online learning with changing action sets, achieving no-approximate-regret guarantees.
problem Online learning with sleeping experts/bandits, where only a subset of actions are available each time.
method Developed computationally efficient algorithms providing no-approximate-regret guarantees for the general problem and better approximation ratios for special cases.
result Achieved no-approximate-regret guarantees for the general sleeping expert/bandit problems and better approximation ratios for specific cases.
Transforms offline algorithms to online with low regret in random order model.
problem Developing online algorithms with low approximate regret from offline approximation algorithms.
method General reduction theorem and coreset construction method.
result Achieves polylogarithmic ε-approximate regret for various online problems.
Algorithm reduces regret in SSP problems with LFA.
problem Finding shortest paths in stochastic environments with linear approximations.
method Uses linear function approximation and stationary policies to minimize regret.
result Achieves sublinear regret under minimal assumptions.
Study improves CTS's approximation regret for combinatorial bandits.
problem Improving CTS's performance on non-exact oracles.
method Develops a new O ( log ( T ) / Δ ) \mathcal{O}(\log(T)/Δ) O ( log ( T ) /Δ ) upper bound for CTS under specific conditions. result First O ( log ( T ) / Δ ) \mathcal{O}(\log(T)/Δ) O ( log ( T ) /Δ ) approximation regret upper bound for CTS. EBUCB framework achieves optimal regret with bounded approximate inference error.
problem Theoretical gap between practical performance and theoretical justification of Bayesian bandit algorithms with approximate inference.
method Enhanced Bayesian Upper Confidence Bound (EBUCB) framework that accommodates bandit problems with approximate inference.
result EBUCB achieves optimal regret order O ( log T ) O(\log T) O ( log T ) under certain conditions on inference error. Logarithmic regret achieved in RL with linear function approximation.
problem Achieving logarithmic regret in reinforcement learning with linear function approximation.
method LSVI-UCB for linear MDP assumption, UCRL-VTR for linear mixture MDP assumption.
result Logarithmic regret bounds established for RL with linear function approximation.
Bayesian bandit algorithms with approximate inference improve regret bounds in stochastic linear bandits.
problem Theoretical justification for Bayesian bandit algorithms with approximate inference in stochastic linear bandits.
method Proposed a theoretical framework to analyze approximate inference impact and conducted frequentist regret analysis on LinTS and LinBUCB.
result LinTS and LinBUCB preserve their original regret upper bounds with larger constant terms in approximate inference settings.
Enhances RL with function approximation, improving regret bounds.
problem Improving exploration in reinforcement learning with function approximation.
method Prior-dependent Bayesian regret bound for PSRL with linear mixture MDPs, using value-targeted model learning and variance reduction.
result Established an upper bound of O ( d H 3 T log T ) {\mathcal{O}}(d\sqrt{H^3 T \log T}) O ( d H 3 T log T ) for PSRL. We argue that the existing regret matchings for Nash equilibrium approximation conduct "jumpy" strategy updating when the probabilities of future plays are set to be proportional to positive regret measures. We propose a geometrical regret matching which features "smooth" strategy updating. Our approach is simple, intu…
VO Q Q Q L optimizes RL with sparse rewards using weighted bounds.
problem Sparse rewards and non-linear function approximation in RL.
method VO Q Q Q L combines Q Q Q -learning with weighted bounds for optimal regret. result Achieves asymptotically optimal regret for linear function approximation.
New algorithm learns LQR with O ( T ) O(\sqrt{T}) O ( T ) regret using Langevin dynamics and excitation.
problem Learning LQR with a O ( T ) O(\sqrt{T}) O ( T ) regret bound. method Thompson sampling with Langevin dynamics and excitation mechanism.
result Achieved O ( T ) O(\sqrt{T}) O ( T ) regret bound for LQR learning. UCRL-WVTR tackles long-term reinforcement learning with general approximations, achieving horizon-free and instance-dependent regret bounds.
problem Long-term reinforcement learning with general function approximations.
method UCRL-WVTR proposes a novel algorithm, UCRL-WVTR, with weighted value-targeted regression and a high-order moment estimator.
result Achieves horizon-free and instance-dependent regret bounds matching minimax lower bounds up to logarithmic factors.
Study on policy gradient for stochastic bandits using diffusion approximation.
problem Improving policy gradient methods for stochastic bandits with optimal regret bounds.
method Continuous-time diffusion approximation of policy gradient with learning rate analysis.
result Proved optimal regret bound of O ( k log ( k ) log ( n ) / η ) O(k \log(k) \log(n) / η) O ( k log ( k ) log ( n ) / η ) for η = O ( Δ 2 / log ( n ) ) η= O(Δ^2/\log(n)) η = O ( Δ 2 / log ( n )) . New algorithms for learning MDPs with linear approximations in infinite-horizon settings.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method Optimism principle, adversarial linear bandits, Natural Policy Gradient.
result Efficient algorithms with optimal or near-optimal regret bounds.
We consider regret minimization in repeated games with non-convex loss functions. Minimizing the standard notion of regret is computationally intractable. Thus, we define a natural notion of regret which permits efficient optimization and generalizes offline guarantees for convergence to an approximate local optimum. W…
Meta-algorithm selects optimal model complexity in RL with function approximation.
problem Selecting the right model complexity for reinforcement learning with function approximation.
method Meta-algorithm that successively rejects complex models using a statistical test.
result Adapts to optimal complexity with improved regret bounds compared to best candidate.
Improved Politex algorithm reduces regret bound to O(√T) with experience replay.
problem Learning in infinite-horizon MDPs with function approximation.
method Sharpened regret analysis of Politex algorithm, experience replay implementation.
result First high-probability O(√T) regret bound for computationally efficient algorithm.
Improved online Q-learning for MDPs with concentration bounds.
problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.
method Smoothed ε n ε_n ε n -Greedy exploration scheme combining ε n ε_n ε n -greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation. result Near- i l d e O ( N 9 / 10 ) ilde{O}(N^{9/10}) i l d e O ( N 9/10 ) regret bound for Smoothed ε n ε_n ε n -Greedy exploration scheme. Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
problem Approximation errors in policy and value function approximations.
method State-aggregated representations and policy gradient methods.
result Policy gradient methods can achieve a per-period regret bounded by ε, while approximate policy iteration and value iteration have a higher regret.
This paper considers the stability of online learning algorithms and its implications for learnability (bounded regret). We introduce a novel quantity called {\em forward regret} that intuitively measures how good an online learning algorithm is if it is allowed a one-step look-ahead into the future. We show that given…
Efficient algorithms for contextual bandits with smooth regret in continuous action spaces.
problem Efficient learning in large or continuous action spaces.
method Smooth regret notion and efficient algorithms for general function approximation.
result Statistically and computationally efficient algorithms for contextual bandits with smooth regret.
New RL algorithm tackles nonstationary MDPs with linear approximations and varying rewards.
problem Nonstationary reinforcement learning with evolving reward and state transition functions.
method Developed a new algorithm LSVI-UCB-Restart with periodic restart, and parameter-free Ada-LSVI-UCB-Restart for unknown variation budgets.
result First minimax dynamic regret lower bound for nonstationary linear MDPs and linear MDPs lower bound.
New online conformal prediction methods minimize strongly adaptive regret and achieve near-optimal coverage.
problem Uncertainty quantification in online settings with changing data distributions.
method Developed new online conformal prediction methods that minimize strongly adaptive regret.
result Achieve near-optimal strongly adaptive regret and approximately valid coverage.
New algorithms reduce contextual bandits' regret without knowing reward noise variances.
problem Reducing regret in contextual bandits with unknown reward noise variances.
method Developed new algorithms based on the optimism principle.
result Regret scales as the square root of the sum of measurement variances, not the time horizon.
Two new algorithms reduce online kernel regression's computational cost while maintaining optimal regret bounds.
problem Trade-off between regret and computational cost in online kernel regression.
method AOGD-ALD and NONS-ALD algorithms dynamically maintain nearly orthogonal basis to approximate kernel mapping and control approximate error.
result Achieves nearly optimal regret bounds at sublinear computational complexity.
New algorithm learns optimal path in reinforcement learning with linear approximations.
problem Optimal path learning in reinforcement learning with linear approximations.
method Proposes novel algorithm with Hoeffding-type and Bernstein-type confidence sets.
result Achieves near-optimal regret guarantee for linear mixture SSP.
ESCHER avoids importance sampling to estimate regret in large games.
problem Estimating Nash equilibria in large games with high variance.
method Computes a history value function to estimate regret without importance sampling.
result ESCHER reduces regret estimation variance significantly compared to existing methods.
Efficient RL algorithms for linear function approximation with limited adaptivity constraints.
problem Limited adaptivity in reinforcement learning with linear function approximation.
method Proposed two efficient online RL algorithms for episodic linear Markov decision processes under batch learning and rare policy switch models.
result Achieved efficient regret bounds for both batch learning and rare policy switch models, with substantial reduction in adaptivity.
New algorithm reduces reinforcement learning regret by adapting to interaction variability.
problem Existing reinforcement learning methods lack adaptability to interaction variability.
method Developed a variance-adaptive optimal algorithm for MNL function approximation.
result Achieved instance-wise optimal regret bounds, validating efficiency in practice.
Improved reinforcement learning algorithm with linear approximation for unknown dynamics.
problem Reinforcement learning with adversarial changing cost functions and bandit feedback.
method Combines mirror-descent and least squares policy evaluation in an auxiliary MDP.
result Obtains an O ~ ( K 6 / 7 ) \widetilde O(K^{6/7}) O ( K 6/7 ) regret bound, significantly improving over previous methods. New algorithms minimize regret in SSP with optimal sparse updates.
problem Minimizing regret in Stochastic Shortest Path models.
method Implicit finite-horizon approximation for analysis, model-free and model-based algorithms developed.
result Minimax optimal regret for both model-free and model-based algorithms.
Improved regret bound for adversarial MDPs with linear function approximation.
problem Learning in adversarial MDPs with changing loss functions and large state spaces.
method Two algorithms: refined FTRL with log-barrier regularizer and magnitude-reduced loss estimator.
result Achieved i l d e O ( K ) ilde{\mathcal O}(\sqrt K) i l d e O ( K ) regret, improving over i l d e O ( K 2 / 3 ) ilde{\mathcal O}(K^{2/3}) i l d e O ( K 2/3 ) . Ensemble sampling approximates Thompson sampling for complex models.
problem Computational intractability of exact posterior distributions.
method Thompson sampling approximation with information-theoretic concepts.
result Established a first rigorous regret bound for ensemble sampling.
We revisit the question of reducing online learning to approximate optimization of the offline problem. In this setting, we give two algorithms with near-optimal performance in the full information setting: they guarantee optimal regret and require only poly-logarithmically many calls to the approximation oracle per it…
Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.
problem Understanding and handling non-stationary environments in reinforcement learning.
method Demonstrated that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings.
result Transformers can approximate and learn strategies for non-stationary environments, matching or outperforming existing expert algorithms.
Papers learn from data to make decisions without interacting, improving on previous methods.
problem Achieving optimal decision-making from offline data with non-linear function approximation.
method Pessimistic Nonlinear Least-Square Value Iteration (PNLSVI) with three innovative components.
result Achieves minimax optimal instance-dependent regret for non-linear function approximation.
LIBO optimizes repeated bandit tasks without prior knowledge or regret.
problem Optimizing repeated bandit tasks without prior knowledge or regret.
method LIBO sequentially meta-learns a kernel to adapt to the environment and solve tasks with the latest estimate.
result LIBO achieves sublinear lifelong regret, converging to oracle performance as more tasks are solved.
Paper proposes OPF policy for fair resource allocation with sublinear regret.
problem Fair resource allocation in an online setting against an unrestricted adversary.
method Online Proportional Fair (OPF) policy achieving approximate sublinear regret.
result OPF policy achieves c α c_α c α -approximate sublinear regret with c α ≤ 1.445 c_α \leq 1.445 c α ≤ 1.445 . New method achieves small-loss regret bounds in random-order model.
problem Online learning with adversarial loss functions in random order.
method Extending batch-to-online transformation, using average sensitivity and stability.
result Small-loss regret bounds of order i l d e O ( φ ⋆ ( O P T T ) ) ilde O(\varphi^{\star}(\mathrm{OPT}_T)) i l d e O ( φ ⋆ ( OPT T )) . Kernel-based function approximation improves reinforcement learning performance.
problem Average reward reinforcement learning in infinite horizon settings.
method Optimistic algorithm based on kernel ridge regression.
result No-regret performance guarantees and confidence intervals for kernel-based predictions.
A new model-free algorithm achieves near-optimal regret for infinite-horizon MDPs.
problem Model-free reinforcement learning for infinite-horizon average-reward MDPs.
method Exploration Enhanced Q-learning (EE-QL) for weakly communicating MDPs.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret bound for general weakly communicating MDPs. We study the effects of approximate inference on the performance of Thompson sampling in the k k k -armed bandit problems. Thompson sampling is a successful algorithm for online decision-making but requires posterior inference, which often must be approximated in practice. We show that even small constant inference error …
Algorithm finds Nash equilibria in complex games with function approximation.
problem Learning Nash equilibria in two-player zero-sum Markov Games with nonlinear function approximation.
method Online learning algorithm using upper and lower confidence bounds derived from optimism in the face of uncertainty.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret with polynomial complexity, under mild assumptions. The paper tackles cooperative RL with function approximation, achieving near-optimal learning with limited communication.
problem Cooperative multi-agent reinforcement learning with function approximation.
method Careful message-passing and cooperative value iteration.
result Achieving near-optimal no-regret learning with limited communication in cooperative multi-agent settings.
This paper analyzes how randomizing rewards in MBRL can improve performance without being overly optimistic.
problem The gap between theoretical worst-case regret analysis and empirical performance in MBRL.
method Reward randomization in model-based reinforcement learning (MBRL) with kernelized linear regulator (KNR) model.
result Reward randomization guarantees partial optimism and near-optimal worst-case regret.
UCRL2-VTR achieves nearly optimal regret for learning MDPs with linear function approximation.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method UCRL2-VTR algorithm with Bernstein-type bonus.
result Achieves a regret of i l d e O ( d D T ) ilde{O}(d\sqrt{DT}) i l d e O ( d D T ) with matching lower bound. New algorithms reduce regret in reinforcement learning with MNL approximations.
problem Efficient reinforcement learning with MNL function approximation for MDPs.
method Proposed randomized exploration algorithms with frequentist regret guarantees.
result Achieved improved regret bounds for MNL transition models.
We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning (RL). When the state space is large or continuous, traditional tabular approaches are unfeasible and some form of function approximation is mandatory. In this paper, we introduce an optimistically-initialized variant of the popula…