On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Upper Confidence Reinforcement Learning”

Recommendation System-based Upper Confidence Bound for Online Advertisingcs.IR

In this paper, the method UCB-RS, which resorts to recommendation system (RS) for enhancing the upper-confidence bound algorithm UCB, is presented. The proposed method is used for dealing with non-stationary and large-state spaces multi-armed bandit problems. The proposed method has been targeted to the problem of the …

2019-09-09abs ↗pdf ↗

Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Losscs.LG

New algorithm reduces regret and constraint violation in adversarial CMDP learning.

problem Online learning for episodic stochastically constrained Markov decision processes (CMDPs) with adversarial loss.

method Upper Confidence Primal-Dual Reinforcement Learning (UC-PDL) algorithm.

result Achieves

\widetilde{\mathcal{O}}(L|\mathcal{S}|\sqrt{|\mathcal{A}|T})

upper bounds of both regret and constraint violation.

Reinforcement Learning CMDP Adversarial Loss Upper Confidence Primal-Dual

2020-03-02abs ↗pdf ↗

Constrained Upper Confidence Reinforcement Learningcs.LG

Constrained Markov Decision Processes are a class of stochastic decision problems in which the decision maker must select a policy that satisfies auxiliary cost constraints. This paper extends upper confidence reinforcement learning for settings in which the reward function and the constraints, described by cost functi…

2020-01-26abs ↗pdf ↗

Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcs.LG

New RL method tackles dynamic MDPs with evolving rewards and states.

problem Dynamic MDPs with evolving rewards and states.

method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).

result Achieves dynamic regret bound for non-stationary MDPs.

Reinforcement Learning Markov Decision Processes Non-Stationarity Dynamic Regret Confidence Widening

2020-06-24abs ↗pdf ↗

Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcs.AI

In the field of reinforcement learning there has been recent progress towards safety and high-confidence bounds on policy performance. However, to our knowledge, no practical methods exist for determining high-confidence policy performance bounds in the inverse reinforcement learning setting---where the true reward fun…

2017-07-03abs ↗pdf ↗

Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Banditscs.LG

New principle optimizes bandit decisions with context.

problem Dealing with general function classes and large context spaces in bandits.

method Upper Counterfactual Confidence Bounds (UCCB) principle.

result Proves optimality and efficiency in complex settings.

bandits optimism context confidence bounds reinforcement learning

2020-07-15abs ↗pdf ↗

Statistically Model Checking PCTL Specifications on Markov Decision Processes via Reinforcement Learningcs.LG

Statistical model checking for PCTL on MDPs using reinforcement learning.

problem Model checking PCTL specifications on MDPs with statistical methods.

method Reinforcement learning for policy search, statistical model checking with UCB-based Q-learning.

result Provably guaranteed statistical model checking method for PCTL specifications on MDPs.

reinforcement learning statistical model checking Markov decision processes probabilistic CTL policy search

2020-04-01abs ↗pdf ↗

Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learningcs.LG

Optimal strategy for reinforcement learning with delayed observations.

problem Delayed state observation in reinforcement learning.

method Combines augmentation method and upper confidence bound approach.

result Minimax optimal regret bound of

ilde{\mathcal{O}}(H \sqrt{D_{\max} SAK})

reinforcement learning delayed observations upper confidence bound minimax optimal tabular MDPs

2026-03-03abs ↗pdf ↗

From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusesstat.ML

Bayes-UCBVI tackles reinforcement learning with a new upper confidence bound method.

problem Optimizing exploration in reinforcement learning without bonuses.

method Bayes-UCBVI uses a quantile of a Q-value function posterior as an upper confidence bound.

result Proves a regret bound of order

\widetilde{O}(\sqrt{H^3SAT})

for tabular reinforcement learning.

reinforcement learning upper confidence bound Bayesian methods tabular settings anti-concentration bound

2022-05-16abs ↗pdf ↗

Combinatorial Rising Banditscs.LG

CRB tackles rising rewards in combinatorial online learning.

problem Rising rewards in combinatorial online learning.

method CRB framework and CRUCB algorithm.

result Empirical and theoretical validation of CRUCB's effectiveness.

rising rewards combinatorial bandits upper confidence bound reinforcement learning sequential decision making

2024-12-01abs ↗pdf ↗

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximationcs.LG

New algorithm reduces RL complexity with low switching costs.

problem Exploration-exploitation dilemma in RL with complex models.

method Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation.

result Achieves minimax optimal regret of

O(d\sqrt{HK})

and near-optimal policy switching cost.

reinforcement learning upper confidence bound function approximation exploration-exploitation minimax optimal

2023-11-26abs ↗pdf ↗

Kernelized Offline Contextual Dueling Banditscs.LG

Efficiently identifies good policies by choosing contexts for human feedback.

problem Efficiently identifying good policies in applications with high feedback costs.

method Introduces offline contextual dueling bandit setting and an upper-confidence-bound style algorithm.

result Proves a regret bound and shows superior performance over uniformly sampled contexts.

contextual dueling bandits human feedback reinforcement learning regret bound upper-confidence-bound

2023-07-21abs ↗pdf ↗

Tightening Exploration in Upper Confidence Reinforcement Learningcs.LG

UCRL3 improves UCRL2's efficiency in reinforcement learning by reducing exploration.

problem Long burn-in phases in numerical experiments of UCRL2.

method UCRL3 uses state-of-the-art time-uniform concentration inequalities and adaptive support computation to tighten exploration.

result UCRL3 achieves a better numerical improvement over UCRL2 in standard environments.

reinforcement learning UCRL2 UCRL3 exploration time-uniform concentration

2020-04-20abs ↗pdf ↗

UCB Exploration via Q-Ensemblescs.LG

We show how an ensemble of $Q^*$ -functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$ -learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experimen…

2017-06-05abs ↗pdf ↗

Learning from an Exploring Demonstrator: Optimal Reward Estimation for Banditsstat.ML

Estimates rewards from a demonstrator's learning process.

problem Estimating rewards from a demonstrator's behavior.

method Leveraging the demonstrator's exploration phase for reward estimation.

result Consistent reward estimation possible without identifiability issues.

reward estimation bandits inverse reinforcement learning exploration demonstrator

2021-06-28abs ↗pdf ↗

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximationstat.ML

Efficient RL algorithm for multinomial logistic MDPs with provable guarantees.

problem Model-based RL for episodic MDPs with unknown transition probabilities.

method Upper confidence bound-based algorithm for exploration-exploitation balance.

result Achieves

ilde{O}(d \sqrt{H^3 T})

regret bound for multinomial logistic models.

Reinforcement Learning Model-Based Multinomial Logistic Upper Confidence Bound Episodic MDPs

2022-12-27abs ↗pdf ↗

Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcs.LG

The paper optimizes risk-sensitive RL with CVaR, achieving near-minimax-optimal results.

problem Optimizing risk-sensitive reinforcement learning with CVaR objective.

method Developed algorithms for multi-arm bandits and online RL in MDPs, achieving near-minimax-optimal regret.

result Achieved near-minimax-optimal regret of

O(τ^{-1}\sqrt{SAK})

for constant

τ

Reinforcement Learning Risk-Sensitive Conditional Value at Risk Upper Confidence Bound Value Iteration

2023-02-07abs ↗pdf ↗

Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Settingcs.LG

LqgOpt learns optimal control in unknown LQG systems with minimal regret.

problem Adaptive control in partially observable linear quadratic Gaussian systems with unknown dynamics.

method Optimism in the face of uncertainty, predictor state evolution, closed-loop system identification, confidence bounds.

result Proves a regret upper bound of

ilde{\mathcal{O}}(\sqrt{T})

for LQG systems.

Reinforcement Learning Optimism Linear Quadratic Gaussian Adaptive Control Uncertainty

2020-03-12abs ↗pdf ↗

When a Reinforcement Learning Agent Encounters Unknown Unknownscs.LG

AI agent learns to handle unknown unknown states in reinforcement learning.

problem Handling unexpected, previously unseen states in reinforcement learning.

method Proposes EMDP-GA model with NIVE approach to expand value functions.

result Asymptotically consistent regret and comparable computational complexity.

reinforcement learning unknown unknowns episodic Markov decision process upper confidence bound momentum Q-learning asymptotically consistent

2025-05-19abs ↗pdf ↗

Non-stationary Reinforcement Learning under General Function Approximationcs.LG

New algorithm tackles non-stationary reinforcement learning with general function approximation.

problem Understanding non-stationary MDPs with function approximation.

method Dynamic Bellman Eluder (DBE) dimension for complexity, sliding window mechanism, confidence set design.

result Upper bound on dynamic regret for proposed SW-OPEA algorithm.

non-stationary reinforcement learning dynamic Bellman Eluder dimension confidence set sliding window dynamic regret

2023-06-01abs ↗pdf ↗

Information-Directed Exploration for Deep Reinforcement Learningcs.LG

Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropri…

2018-12-18abs ↗pdf ↗

Logarithmic regret bounds for continuous-time average-reward Markov decision processescs.LG

Logarithmic regret for continuous-time reinforcement learning.

problem Continuous-time Markov decision processes with unknown transition probabilities and holding times.

method Upper confidence reinforcement learning, mean holding time estimation, stochastic comparison of point processes.

result Logarithmic regret bound achieved in finite time.

reinforcement learning continuous-time MDPs average-reward logarithmic regret stochastic comparison

2022-05-23abs ↗pdf ↗

UCB Momentum Q-learning: Correcting the bias without forgettingstat.ML

UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.

problem Improving Q-learning's bias and regret in reinforcement learning.

method UCBMQ combines Q-learning with an upper confidence bound and momentum term.

result UCBMQ guarantees a regret of

O(\sqrt{H^3SAT}+ H^4 S A )

with a linear second-order term in

S

reinforcement learning Q-learning momentum upper confidence bound regret

2021-03-01abs ↗pdf ↗

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertaintycs.LG

Paper develops a new RL method for MDPs with uncertainty, achieving better regret bounds.

problem Online reinforcement learning in environments with both endogenous and exogenous uncertainty.

method Developed a VB-UCRL algorithm that restarts based on variation schedules.

result Established a regret bound of saving at most

\sqrt{S}

S^{\frac{1}{6}}T^{\frac{1}{12}}

Reinforcement Learning Markov Decision Processes Uncertainty Regret Bound Variation Aware

2021-06-15abs ↗pdf ↗

Model Selection for Generic Reinforcement Learningstat.ML

ARL-GEN adapts to the smallest model class in nested families for RL with improved regret.

problem Model selection for Reinforcement Learning with nested model families.

method Adaptive Reinforcement Learning (ARL-GEN) with value targeted regression and model selection module.

result ARL-GEN achieves a matching regret to an oracle with knowledge of the true model class.

Reinforcement Learning Model Selection Upper Confidence Reinforcement Learning UCRL Eluder Dimension

2021-07-13abs ↗pdf ↗

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Banditscs.LG

A new UCB algorithm for heavy-tailed bandits with near-optimal regret.

problem Sequential decision making in uncertain environments with heavy-tailed rewards.

method Data-driven, distribution-free UCB algorithm combining resampled median-of-means and UCB.

result Near-optimal regret bound for heavy-tailed distributions.

UCB algorithm bandits heavy-tailed resampled median-of-means stochastic multi-armed

2024-06-09abs ↗pdf ↗

Non-Stationary Reinforcement Learning: The Blessing of (More) Optimismcs.LG

We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs) under temporal drifts, ie, both the reward and state transition distributions are allowed to evolve over time, as long as their respective total variations, quantified by suitable metrics, do not exceed certain variation budgets. …

2019-06-07abs ↗pdf ↗

Exploration via Epistemic Value Estimationcs.LG

Proposes EVE for efficient exploration in reinforcement learning.

problem Efficient exploration in reinforcement learning.

method EVE: a recipe for posterior over parameters, facilitating efficient exploration.

result Competitive performance on benchmarks, efficient exploration confirmed.

reinforcement learning exploration epistemic value sequential decision making neural networks

2023-03-07abs ↗pdf ↗

Reinforcement Learning with History-Dependent Dynamic Contextscs.LG

New framework handles dynamic contexts in reinforcement learning.

problem Learning in environments where contexts change over time.

method Dynamic Contextual Markov Decision Processes (DCMDPs) with logistic aggregation.

result Upper-confidence-bound style algorithm with regret bounds.

Reinforcement Learning Dynamic Contexts Markov Decision Processes Optimism Latent Space Planning

2023-02-04abs ↗pdf ↗

CoinDICE: Off-Policy Confidence Interval Estimationcs.LG

CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.

problem Estimating value of a target policy using only behavior policy data.

method Function space embedding, generalized empirical likelihood method, Lagrangian optimization.

result Valid confidence intervals with tighter and more accurate estimates than existing methods.

reinforcement learning off-policy evaluation confidence intervals generalized empirical likelihood behavior policies

2020-10-22abs ↗pdf ↗

Learning Stochastic Shortest Path with Linear Function Approximationcs.LG

New algorithm learns optimal path in reinforcement learning with linear approximations.

problem Optimal path learning in reinforcement learning with linear approximations.

method Proposes novel algorithm with Hoeffding-type and Bernstein-type confidence sets.

result Achieves near-optimal regret guarantee for linear mixture SSP.

reinforcement learning stochastic shortest path linear function approximation regret minimization confidence sets

2021-10-25abs ↗pdf ↗

Near-optimal Reinforcement Learning in Factored MDPsstat.ML

Any reinforcement learning algorithm that applies to all Markov decision processes (MDPs) will suffer $Ω(\sqrt{SAT})$ regret on some MDP, where $T$ is the elapsed time and $S$ and $A$ are the cardinalities of the state and action spaces. This implies $T = Ω(SA)$ time to guarantee a near-optimal policy. In many settings…

2014-03-15abs ↗pdf ↗

Monte-Carlo Tree Search for Policy Optimizationcs.LG

Gradient-based methods are often used for policy optimization in deep reinforcement learning, despite being vulnerable to local optima and saddle points. Although gradient-free methods (e.g., genetic algorithms or evolution strategies) help mitigate these issues, poor initialization and local optima are still concerns …

2019-12-23abs ↗pdf ↗

Prior-dependent analysis of posterior sampling reinforcement learning with function approximationstat.ML

Enhances RL with function approximation, improving regret bounds.

problem Improving exploration in reinforcement learning with function approximation.

method Prior-dependent Bayesian regret bound for PSRL with linear mixture MDPs, using value-targeted model learning and variance reduction.

result Established an upper bound of

{\mathcal{O}}(d\sqrt{H^3 T \log T})

for PSRL.

reinforcement learning function approximation PSRL Bayesian regret linear mixture MDPs

2024-03-17abs ↗pdf ↗

Quantum Computing Provides Exponential Regret Improvement in Episodic Reinforcement Learningcs.LG

Quantum UCB algorithm reduces reinforcement learning regret exponentially.

problem Episodic reinforcement learning with quantum state evolution.

method Upper Confidence Bound (UCB) quantum algorithm with quantum mean estimation.

result Exponential improvement in regret from $\Tilde{\mathcal{O}}(\sqrt{K})$ to $\Tilde{\mathcal{O}}(1)$.

Quantum Computing Reinforcement Learning UCB Algorithm Regret Minimization Episodic MDP

2023-02-16abs ↗pdf ↗

On Lai's Upper Confidence Bound in Multi-Armed Banditsstat.ML

The paper honors Lai's contributions to multi-armed bandits and establishes new regret bounds.

problem Improving regret bounds in multi-armed bandit problems.

method Establishes non-asymptotic regret bounds for upper confidence bound indices.

result New regret bounds match Lai-Robbins lower bound.

multi-armed bandits upper confidence bound regret bounds Tze Leung Lai

2024-10-03abs ↗pdf ↗

Learning to Switch Among Agents in a Team via 2-Layer Markov Decision Processescs.LG

Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions. In this work, our goal is to develop algorithms that, by learning to switch control between agents, allow existing reinforcement learning agents …

2020-02-11abs ↗pdf ↗

Differentiable Linear Bandit Algorithmcs.LG

A new differentiable UCB algorithm for linear bandits learns adaptive confidence bounds.

problem Inability of UCB to strike optimal exploration-exploitation due to confidence bounds.

method Proposes a differentiable linear bandit algorithm and a gradient estimator for learning adaptive confidence bounds.

result Achieves a

ilde{\mathcal{O}}(\hatβ\sqrt{dT})

upper bound of

T

-round regret.

bandit algorithms confidence bounds differentiability gradient ascent regret minimization

2020-06-04abs ↗pdf ↗

Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcs.LG

RH-UCRL combines pessimism and optimism for robust RL.

problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.

method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.

result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.

robust reinforcement learning model-based RL worst-case optimization neural networks ensemble models

2021-03-18abs ↗pdf ↗

KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systemscs.LG

KCRL learns stable policies for nonlinear systems with formal guarantees.

problem Lack of stabilization guarantees in RL methods for safety-critical systems.

method KCRL uses Krasovskii's Lyapunov functions as a stability constraint and a primal-dual approach to learn stabilizing policies.

result KCRL guarantees learning a stabilizing policy in a finite number of interactions.

Reinforcement Learning Stability Guarantees Nonlinear Systems Lyapunov Functions Policy Optimization

2022-06-03abs ↗pdf ↗

Randomized Exploration for Reinforcement Learning with General Value Function Approximationcs.LG

New algorithm explores reinforcement learning with noisy data.

problem Exploration in reinforcement learning with complex value functions.

method Randomized exploration with i.i.d. scalar noises and optimistic reward sampling.

result Achieves worst-case regret bound of

\widetilde{O}(\mathrm{poly}(d_EH)\sqrt{T})

reinforcement learning exploration value function approximation UCB regret bound

2021-06-15abs ↗pdf ↗

Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscs.LG

Federated UCBVI reduces communication costs while minimizing regret in multi-agent settings.

problem Minimizing regret in federated learning with heterogeneous agents.

method Federated Upper Confidence Bound Value Iteration (Fed-UCBVI) algorithm.

result Regret bound scales as

ilde{\mathcal{O}}(\sqrt{H^3 |\mathcal{S}| |\mathcal{A}| T / M})

with small additional term due to heterogeneity.

Federated Learning Reinforcement Learning UCBVI Heterogeneity Communication Efficiency

2024-10-30abs ↗pdf ↗

Exact Minimum-Volume Confidence Set Intersection for Multinomial Outcomesstat.ML

Paper certifies intersection of minimum-volume confidence sets for multinomial outcomes.

problem Certifying intersection of minimum-volume confidence sets for multinomial outcomes.

method Exploits likelihood ordering to induce halfspace constraints, enabling adaptive geometric partitioning and computable bounds on p-values.

result Efficient and provably sound algorithm for certifying intersection, disjointness, or indeterminate result.

confidence sets multinomial outcomes minimum-volume intersection certified decision

2026-01-26abs ↗pdf ↗

Open Problem: Tight Online Confidence Intervals for RKHS Elementsstat.ML

Research aims to improve confidence intervals for RKHS elements in online learning.

problem Improper confidence intervals lead to suboptimal regret bounds in kernel-based bandit and reinforcement learning.

method Formalizes the open problem of online confidence intervals in RKHS and reviews existing results.

result Identifies the online nature of observation points as the main challenge for tight confidence intervals.

confidence intervals reproducing kernel Hilbert space online learning kernel bandit reinforcement learning

2021-10-28abs ↗pdf ↗

Provably Efficient CVaR RL in Low-rank MDPscs.LG

New RL algorithm maximizes CVaR in low-rank MDPs with provable efficiency.

problem Maximizing CVaR in large state spaces with function approximation.

method Upper Confidence Bound (UCB) bonus-driven algorithm for low-rank MDPs.

result Achieves sample complexity of O(H^7 A^2 d^4 / τ^2 ε^2) for ε-optimal CVaR.

Reinforcement Learning Conditional Value at Risk Low-rank MDPs Upper Confidence Bound Function Approximation

2023-11-20abs ↗pdf ↗

Learning Fair Representations with High-Confidence Guaranteescs.LG

Paper defines and solves a problem in representation learning to ensure fairness with high confidence.

problem Learning fair representations with high confidence guarantees for all downstream tasks.

method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.

result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.

representation learning fairness high-confidence guarantees downstream tasks downstream models

2023-10-23abs ↗pdf ↗

Instance-Dependent Confidence and Early Stopping for Reinforcement Learningstat.ML

The research proposes a stopping rule for reinforcement learning algorithms based on instance-dependent confidence.

problem Dramatic variation in convergence rates of reinforcement learning algorithms due to problem structure.

method Develops instance-dependent confidence regions and a data-dependent stopping rule for MDP policy evaluation and optimal value estimation.

result Proposes a stopping rule that adapts to the instance-specific difficulty of the problem, allowing for early termination.

reinforcement learning instance-dependent confidence regions early stopping policy evaluation

2022-01-21abs ↗pdf ↗

Provable Self-Play Algorithms for Competitive Reinforcement Learningcs.LG

Self-play, where the algorithm learns by playing against itself without requiring any direct supervision, has become the new weapon in modern Reinforcement Learning (RL) for achieving superhuman performance in practice. However, the majority of exisiting theory in reinforcement learning only applies to the setting wher…

2020-02-10abs ↗pdf ↗