On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “episodic reinforcement”

Constrained episodic reinforcement learning in concave-convex and knapsack settingscs.LG

Algorithm tackles constrained reinforcement learning with concave-convex and knapsack constraints.

problem Constrained episodic reinforcement learning with concave rewards and convex constraints.

method Modular analysis with strong theoretical guarantees for concave-convex and knapsack settings.

result Significantly outperforms existing approaches in constrained episodic environments.

reinforcement learning constrained optimization episodic concave-convex knapsack

2020-06-09abs ↗pdf ↗

Learning Kernel-Based MDPs from Episodic Preferential Feedbackstat.ML

Study preference-based reinforcement learning in episodic kernel MDPs.

problem Learning from episodic human preferences in reinforcement learning.

method Developed preference-based value estimation and confidence sets for kernel-based MDPs.

result Proved high-probability regret bounds that converge to optimal policy value.

reinforcement learning human feedback episodic MDPs kernel methods regret bounds

2026-05-22abs ↗pdf ↗

Learning Long-Term Reward Redistribution via Randomized Return Decompositioncs.LG

Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.

problem Learning from sparse and delayed rewards in reinforcement learning.

method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.

result Substantial improvement over baseline algorithms in experiments.

reinforcement learning sparse rewards delayed rewards proxy reward episodic reinforcement

2021-11-26abs ↗pdf ↗

Towards sample-efficient episodic control with DAC-MLcs.AI

Paper proposes DAC-ML, a cognitive architecture that learns quickly from few episodes.

problem Sample inefficiency in AI learning action policies.

method Incorporates hippocampus-inspired sequential memory system into DAC theory of mind.

result DAC-ML rapidly converges to effective action policies maximizing reward.

sample inefficiency episodic control Deep Reinforcement Learning DAC theory of mind sequential memory

2020-12-26abs ↗pdf ↗

Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?cs.LG

Long horizon reinforcement learning is as hard as short horizon learning.

problem Understanding the difficulty of long horizon reinforcement learning problems.

method Introduced new concepts: ε-net for optimal policies and Online Trajectory Synthesis algorithm.

result Proved that sample complexity scales logarithmically with the planning horizon, refuting the conjecture.

reinforcement learning horizon sample complexity episodic policy

2020-05-01abs ↗pdf ↗

Quantum Computing Provides Exponential Regret Improvement in Episodic Reinforcement Learningcs.LG

Quantum UCB algorithm reduces reinforcement learning regret exponentially.

problem Episodic reinforcement learning with quantum state evolution.

method Upper Confidence Bound (UCB) quantum algorithm with quantum mean estimation.

result Exponential improvement in regret from $\Tilde{\mathcal{O}}(\sqrt{K})$ to $\Tilde{\mathcal{O}}(1)$.

Quantum Computing Reinforcement Learning UCB Algorithm Regret Minimization Episodic MDP

2023-02-16abs ↗pdf ↗

Tighter Regret Bounds for Contextual Action-Set Reinforcement Learningcs.LG

Improved reinforcement learning for episodes with varying action sets.

problem Reinforcement learning with context-dependent action sets.

method Extends MVP algorithm to handle adversarial and stochastic contexts.

result Established minimax regret bounds of

O(\sqrt{SAH^3K\log L})

for adversarial contexts.

Reinforcement Learning Contextual Action-Set Adversarial Stochastic Regret Bounds

2026-05-15abs ↗pdf ↗

Ecological Reinforcement Learningcs.LG

This paper explores how environmental properties can simplify reinforcement learning in non-episodic settings.

problem Challenges in reinforcement learning with continuous interaction and sparse delayed rewards.

method Analysis of environment shaping and dynamism properties to simplify learning.

result Properties like environment shaping and dynamism can significantly ease learning in non-episodic, sparse reward settings.

reinforcement learning non-episodic sparse rewards environment shaping dynamism

2020-06-22abs ↗pdf ↗

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecs.LG

We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states.…

2018-05-31abs ↗pdf ↗

Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncs.LG

UCB-Advantage learns MDPs with $O(\sqrt{H^2SAT})$ regret.

problem Model-free reinforcement learning in finite-horizon MDPs.

method Reference-Advantage decomposition for low regret.

result Achieves

ilde{O}(\sqrt{H^2SAT})

regret, matching best known bounds.

reinforcement learning finite-horizon MDPs UCB-Advantage regret bound model-free

2020-04-21abs ↗pdf ↗

Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learningcs.LG

Recent advances in deep reinforcement learning algorithms have shown great potential and success for solving many challenging real-world problems, including Go game and robotic applications. Usually, these algorithms need a carefully designed reward function to guide training in each time step. However, in real world, …

2019-05-31abs ↗pdf ↗

Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncs.LG

New algorithm reduces reinforcement learning complexity, approaching contextual bandits.

problem Episodic reinforcement learning's difficulty compared to contextual bandits.

method Proposes MVP algorithm with a new Bernstein-type bonus for episodic reinforcement learning.

result Achieves near-optimal regret bound of $O\left(\left(\sqrt{SAK} + S^2A ight) \poly\log \left(SAHK ight) ight)$, improving state-of-the-art results.

reinforcement learning bandits episodic contextual regret

2020-09-28abs ↗pdf ↗

On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcs.LG

Study shows reinforcement learning is possible with once-per-episode feedback.

problem Challenges of reinforcement learning with sparse feedback.

method Introduced a model where trajectory labels are generated by an unknown parametric model and developed an algorithm for sublinear regret.

result Achieved sublinear regret with a statistically and computationally efficient algorithm.

reinforcement learning sparse feedback sublinear regret trajectory labels parametric model

2021-05-29abs ↗pdf ↗

Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Controlcs.LG

Deep networks have enabled reinforcement learning to scale to more complex and challenging domains, but these methods typically require large quantities of training data. An alternative is to use sample-efficient episodic control methods: neuro-inspired algorithms which use non-/semi-parametric models that predict valu…

2019-11-21abs ↗pdf ↗

Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learningstat.ML

We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the mode…

2017-05-09abs ↗pdf ↗

(More) Efficient Reinforcement Learning via Posterior Samplingstat.ML

Most provably-efficient learning algorithms introduce optimism about poorly-understood states and actions to encourage exploration. We study an alternative approach for efficient exploration, posterior sampling for reinforcement learning (PSRL). This algorithm proceeds in repeated episodes of known duration. At the sta…

2013-06-04abs ↗pdf ↗

Model-Free Episodic Controlstat.ML

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and…

2016-06-14abs ↗pdf ↗

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscs.LG

StepMix algorithm ensures safe exploration in reinforcement learning with near-optimal performance.

problem Conservative exploration in reinforcement learning under episode-wise constraints.

method StepMix algorithm balances exploitation and exploration while ensuring episode-wise conservative constraint.

result StepMix achieves near-optimal regret order as in the constraint-free setting.

reinforcement learning conservative exploration regret Markov Decision Process episodic

2023-06-09abs ↗pdf ↗

Optimistic Q-learning for average reward and episodic reinforcement learningcs.LG

New algorithm for average reward learning with bounded hitting time assumption.

problem Minimizing regret in average reward reinforcement learning with bounded hitting time.

method Optimistic Q-learning with a novel

\overline{L}

operator for bounded hitting time.

result Regret bound of

ilde{O}(H^5 S\sqrt{AT})

for average reward learning.

reinforcement learning average reward bounded hitting time Q-learning episodic

2024-07-18abs ↗pdf ↗

Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizonmath.OC

Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.

problem Optimizing control actions in unknown continuous-time systems over a finite time horizon.

method Least-squares algorithm based on continuous-time observations and controls, with perturbation analysis and parameter estimation error analysis.

result Logarithmic regret bound of order

O((\ln M)(\ln\ln M))

reinforcement learning linear-quadratic continuous-time logarithmic regret finite-time horizon

2020-06-27abs ↗pdf ↗

Optimism in Reinforcement Learning with Generalized Linear Function Approximationstat.ML

We design a new provably efficient algorithm for episodic reinforcement learning with generalized linear function approximation. We analyze the algorithm under a new expressivity assumption that we call "optimistic closure," which is strictly weaker than assumptions from prior analyses for the linear setting. With opti…

2019-12-09abs ↗pdf ↗

Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Losscs.LG

New algorithm reduces regret and constraint violation in adversarial CMDP learning.

problem Online learning for episodic stochastically constrained Markov decision processes (CMDPs) with adversarial loss.

method Upper Confidence Primal-Dual Reinforcement Learning (UC-PDL) algorithm.

result Achieves

\widetilde{\mathcal{O}}(L|\mathcal{S}|\sqrt{|\mathcal{A}|T})

upper bounds of both regret and constraint violation.

Reinforcement Learning CMDP Adversarial Loss Upper Confidence Primal-Dual

2020-03-02abs ↗pdf ↗

A PAC RL Algorithm for Episodic POMDPscs.LG

Many interesting real world domains involve reinforcement learning (RL) in partially observable environments. Efficient learning in such domains is important, but existing sample complexity bounds for partially observable RL are at least exponential in the episode length. We give, to our knowledge, the first partially …

2016-05-25abs ↗pdf ↗

Neural Episodic Controlcs.LG

Deep reinforcement learning methods attain super-human performance in a wide range of environments. Such methods are grossly inefficient, often taking orders of magnitudes more data than humans to achieve reasonable performance. We propose Neural Episodic Control: a deep reinforcement learning agent that is able to rap…

2017-03-06abs ↗pdf ↗

A Unifying View of Optimism in Episodic Reinforcement Learningcs.LG

Unified framework for model- and value-optimistic reinforcement learning.

problem Designing and analyzing algorithms for episodic reinforcement learning.

method Lagrangian duality for a unified view of optimism.

result Unified algorithms with both efficient implementation and probabilistic analysis.

reinforcement learning optimism episodic dynamic programming Lagrangian duality

2020-07-03abs ↗pdf ↗

Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guaranteesstat.ML

OPSRL algorithm reduces regret with few samples in reinforcement learning.

problem High regret in reinforcement learning with limited data.

method Optimistic Posterior Sampling (OPSRL) with logarithmic sample complexity.

result Guaranteed high-probability regret bound of

\widetilde{\mathcal{O}}(\sqrt{H^3SAT})

reinforcement learning posterior sampling regret minimization Markov decision process few samples

2022-09-28abs ↗pdf ↗

Reinforcement Learning with Action-Triggered Observationscs.LG

New framework for reinforcement learning with sporadic state observations.

problem Partial observability in reinforcement learning.

method Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs).

result Optimistic algorithm achieving regret bound for episodic learning.

Reinforcement Learning Partial Observability Sporadic Observations Markov Decision Processes Optimistic Algorithms

2025-10-02abs ↗pdf ↗

Corruption-robust exploration in episodic reinforcement learningcs.LG

We initiate the study of multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system extending recent results for the special case of stochastic bandits. We provide a framework which modifies the aggressive exploration enjoyed b…

2019-11-20abs ↗pdf ↗

Exponential Family Model-Based Reinforcement Learning via Score Matchingcs.LG

SMRL uses score matching for efficient RL with exponential family models.

problem Efficient reinforcement learning with unknown transition models.

method Score matching for parameter estimation of exponential family models.

result Achieves online regret of

O(d\sqrt{H^3T})

reinforcement learning score matching exponential family regret bound

2021-12-28abs ↗pdf ↗

A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescs.LG

KeRNS tackles non-stationary reinforcement learning in metric spaces.

problem Non-stationary reinforcement learning in metric spaces.

method KeRNS uses time-dependent kernels to model non-stationary Markov Decision Processes (MDPs).

result KeRNS achieves a regret bound that scales with the covering dimension and total variation of the MDP.

reinforcement learning non-stationary metric spaces kernel-based episodic

2020-07-09abs ↗pdf ↗

Posterior Sampling for Reinforcement Learning Without Episodesstat.ML

This is a brief technical note to clarify some of the issues with applying the application of the algorithm posterior sampling for reinforcement learning (PSRL) in environments without fixed episodes. In particular, this paper aims to: - Review some of results which have been proven for finite horizon MDPs (Osband et a…

2016-08-09abs ↗pdf ↗

Safe Reinforcement Learning with Linear Function Approximationcs.LG

Safe reinforcement learning tackles safety constraints with linear approximations.

problem Ensuring safety in reinforcement learning without violating constraints.

method Modeling safety as a linear cost function, developing SLUCB-QVI and RSLUCB-QVI algorithms for MDPs with linear function approximation.

result Achieved a nearly optimal regret bound for safe reinforcement learning, matching state-of-the-art unsafe algorithms.

reinforcement learning safety constraints linear function approximation regret bound episodic MDPs

2021-06-11abs ↗pdf ↗

BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settingscs.LG

Extends Thompson sampling for RL with fewer episodes.

problem Limited episodes in RL settings.

method Batch Bayesian optimization over episodes to learn action bias terms.

result Significantly outperforms standard Thompson sampling.

Bayesian optimization Thompson sampling Reinforcement learning Episode-limited Health interventions

2024-11-30abs ↗pdf ↗

Reinforcement Learning of POMDPs using Spectral Methodscs.AI

We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…

2016-02-25abs ↗pdf ↗

Posterior Sampling-based Online Learning for Episodic POMDPscs.LG

PS4POMDPs algorithm simplifies online learning for episodic POMDPs with unknown models.

problem Learning in POMDPs is harder than in MDPs; online learning is especially challenging.

method Posterior Sampling-based reinforcement learning algorithm (PS4POMDPs)

result Bayesian regret scales as √number of episodes and is polynomial in other parameters.

POMDPs online learning reinforcement learning Bayesian regret multi-agent

2023-10-16abs ↗pdf ↗

Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcs.LG

This paper studies poisoning attacks in episodic RL and discovers their effectiveness depends on reward bounds.

problem Understanding security threats to RL algorithms through poisoning attacks.

method Examined two types of poisoning attacks: reward and action manipulation, in bounded and unbounded reward settings.

result The effectiveness of poisoning attacks depends on reward bounds, with different attack costs and success rates.

poisoning attacks reinforcement learning episodic reward manipulation action manipulation

2022-08-29abs ↗pdf ↗

Online Episodic Convex Reinforcement Learningcs.LG

New algorithm for online learning in episodic MDPs with convex objectives.

problem Online episodic convex reinforcement learning.

method Online mirror descent algorithm with varying constraint sets and exploration bonus.

result Near-optimal regret bounds for online CURL without prior knowledge of transition function.

online learning reinforcement learning convex optimization mirror descent regret bounds

2025-05-12abs ↗pdf ↗

Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetsstat.ML

UDRL fails to converge in stochastic environments with episodic resets.

problem UDRL's convergence in stochastic environments with resets is questioned.

method UDRL is a supervised learning approach that does not use value functions.

result UDRL diverges in a simple stochastic environment with resets.

Reinforcement Learning Supervised Learning Episodic Resets Convergence Failure UDRL

2022-05-13abs ↗pdf ↗

Uniformly Conservative Exploration in Reinforcement Learningcs.LG

Proposes a method to avoid excessive exploration in reinforcement learning.

problem Avoiding excessive exploration in reinforcement learning to deploy it in practice.

method Designs a novel algorithm using UCB reinforcement learning policy with adaptive exploration constraints.

result Proves that the approach remains conservative while minimizing regret in tabular settings and validates on real-world tasks.

reinforcement learning exploration conservatism UCB regret

2021-10-25abs ↗pdf ↗

Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescs.LG

Study risk-sensitive reinforcement learning with Lipschitz dynamic risk measures, establishing regret bounds.

problem Risk-sensitive reinforcement learning in Markov decision processes.

method Two model-based algorithms for Lipschitz dynamic risk measures, focusing on regret bounds.

result Upper bounds demonstrate optimal dependencies on actions and episodes, reflecting risk sensitivity vs. sample complexity trade-off.

reinforcement learning risk sensitivity dynamic risk measures regret bounds Markov decision processes

2023-06-04abs ↗pdf ↗

Near-optimal Policy Optimization Algorithms for Learning Adversarial Linear Mixture MDPscs.LG

Paper optimizes learning MDPs with adversarial rewards using linear models.

problem Learning MDPs with adversarial rewards and changing reward functions.

method Proposes POWERS, an optimistic policy optimization algorithm.

result Achieves nearly optimal regret bound of

O(dH\sqrt{T})

policy optimization reinforcement learning MDPs adversarial rewards linear models

2021-02-17abs ↗pdf ↗

Episodic Memory Deep Q-Networkscs.LG

Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently,…

2018-05-19abs ↗pdf ↗

Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscs.LG

New framework for RL with linear-convex models reduces performance gap.

problem Continuous-time episodic reinforcement learning with unknown coefficients and convex objectives.

method Probabilistic framework and phase-based learning algorithm for optimal exploration-exploitation trade-off.

result Sublinear regrets achieved, matching best possible results in literature.

reinforcement learning episodic continuous-time linear-convex performance gap

2021-12-19abs ↗pdf ↗

Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learningcs.LG

Study gap-dependent regret bounds for risk-sensitive RL.

problem Risk-sensitive reinforcement learning with entropic risk measure.

method Propose cascaded gaps to adapt to problem structures, derive regret bounds.

result Exponential improvement over existing bounds in appropriate settings.

reinforcement learning risk-sensitive gap-dependent regret episodic MDP

2022-03-07abs ↗pdf ↗

Settling the Horizon-Dependence of Sample Complexity in Reinforcement Learningcs.LG

New algorithm learns optimal policies with just 1 episode, settling horizon-dependence in RL.

problem Understanding the sample complexity of reinforcement learning with horizon length.

method Developed an algorithm using only

O(1)

episodes to achieve PAC guarantee, leveraging connections between value functions in discounted and finite-horizon MDPs and novel perturbation analysis.

result Achieved the same PAC guarantee with only

O(1)

episodes of environment interactions, completely settling horizon-dependence in RL.

reinforcement learning sample complexity horizon dependence value functions MDPs

2021-11-01abs ↗pdf ↗

Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcs.LG

Paper establishes new lower bounds for MDPs with changing transition kernels.

problem Minimizing sample complexity and regret in non-stationary MDPs.

method Developed novel lower bounds and constructed hard MDPs.

result Proved

Ω((H^3SA/ε^2)\log(1/δ))

sample complexity lower bound.

Reinforcement Learning MDP Non-stationary Lower Bounds PAC

2020-10-07abs ↗pdf ↗

Memory-Efficient Episodic Control Reinforcement Learning with Dynamic Online k-meanscs.LG

Recently, neuro-inspired episodic control (EC) methods have been developed to overcome the data-inefficiency of standard deep reinforcement learning approaches. Using non-/semi-parametric models to estimate the value function, they learn rapidly, retrieving cached values from similar past states. In realistic scenarios…

2019-11-21abs ↗pdf ↗

Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescs.LG

New RL algorithm achieves nearly optimal performance for linear MDPs.

problem Optimal reinforcement learning for episodic linear MDPs.

method Weighted linear regression with variance estimator and rare-switching policy.

result Achieves nearly minimax optimal regret

ilde O(d\sqrt{H^3K})

reinforcement learning linear MDPs regret minimization weighted regression variance estimator

2022-12-12abs ↗pdf ↗