Many interesting real world domains involve reinforcement learning (RL) in partially observable environments. Efficient learning in such domains is important, but existing sample complexity bounds for partially observable RL are at least exponential in the episode length. We give, to our knowledge, the first partially …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “partial episodes”
New method learns near-optimal policies with polynomial samples in A and H.
problem Episodic latent MABs with partial observations are challenging.
method Experiment design and method-of-moments approach.
result Polynomial samples in A and H for near-optimal policy learning.
New method trains GFlowNets from partial episodes to improve convergence and stability.
problem Improving convergence and stability of GFlowNets training.
method Introducing SubTB() for GFlowNet training from partial action subsequences.
result SubTB() accelerates GFlowNet convergence and enables training in longer action sequences.
Study evaluates policies in partially observable environments without full model specification.
problem Evaluating policies in partially observable environments without full model specification.
method Developed non-parametric identification and recursive fitted-Q-evaluation algorithm.
result Established finite-sample error bounds for policy value estimation.
We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual traject…
This work improves policy-based training by proposing an evaluation balance objective for GFlowNets.
problem Reliable estimation of policy divergence under directed acyclic graphs remains challenging.
method Proposes an evaluation balance objective over partial episodes to measure policy divergence and improve policy-based training reliability.
result Evaluation balance strengthens policy-based training reliability and broadens its flexibility.
New framework for reinforcement learning with sporadic state observations.
problem Partial observability in reinforcement learning.
method Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs).
result Optimistic algorithm achieving regret bound for episodic learning.
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…
New algorithms improve privacy in bandit problems with partial information.
problem Privacy constraints in multi-armed bandit problems with partial reward information.
method Proposed a generic framework for designing -global DP extensions of UCB and KL-UCB algorithms.
result AdaP-KLUCB algorithm achieves optimal regret bound under -global DP constraints.
Optimistic Q-learning reduces sample complexity for systems with known dynamics.
problem Sample efficiency in reinforcement learning with partial dynamics knowledge.
method Optimistic Q-learning algorithm for systems with additive disturbance model.
result Achieves regret under perfect knowledge of dynamics.
Study preference-based reinforcement learning in episodic kernel MDPs.
problem Learning from episodic human preferences in reinforcement learning.
method Developed preference-based value estimation and confidence sets for kernel-based MDPs.
result Proved high-probability regret bounds that converge to optimal policy value.
Algorithm tackles constrained reinforcement learning with concave-convex and knapsack constraints.
problem Constrained episodic reinforcement learning with concave rewards and convex constraints.
method Modular analysis with strong theoretical guarantees for concave-convex and knapsack settings.
result Significantly outperforms existing approaches in constrained episodic environments.
Algorithm improves multi-armed bandit performance by transferring reward samples.
problem Sequential multi-armed bandit problem with changing reward distributions.
method UCB algorithm with reward sample transfer.
result Significant improvement in cumulative regret over standard UCB.
BerlinUCB learns from episodic rewards in nonstationary contexts.
problem Online learning with episodic rewards in nonstationary environments.
method BerlinUCB integrates clustering for self-supervision.
result BerlinUCB outperforms standard contextual bandits in various scenarios.
We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show regret bound, where is the number of episodes, is the state space, $A…
New algorithm learns POMDPs with known observation model efficiently.
problem Learning POMDPs with unknown transition model in average-reward setting.
method OAS estimation technique and OAS-UCRL algorithm balancing exploration-exploitation.
result Regret guarantee of order for OAS-UCRL algorithm.
A method for self-supervised representation learning in partially observable environments.
problem Sparse rewards and stochasticity in partially observable environments.
method World model in latent space for estimating missing information.
result Significant improvement in exploration compared to prior work.
Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.
problem Memory models struggle in partially observable reinforcement learning environments.
method Introduces a novel memory model using the Hadamard product for efficient memory management and updates.
result Significantly outperforms state-of-the-art memory-based methods on challenging benchmarks.
The paper develops algorithms for competitive RL in partially observable MGs.
problem Challenges in reinforcement learning with function approximation and partial observability.
method Proposes posterior sampling methods for self-play and adversarial learning in zero-sum MGs.
result Developed algorithms achieve low regret bounds scaling sublinearly with GEC and episode number.
Study efficient reinforcement learning for partially observed systems with linear structure.
problem Efficient reinforcement learning for partially observed Markov decision processes with linear structure.
method Proposes OP-TENET algorithm using a Bellman operator with finite memory, adversarial integral equation, and optimistic exploration.
result Achieves ε-optimal policy within O(1/ε^2) episodes with polynomial sample complexity in intrinsic dimension.
Algorithm reduces episode count for CMDPs with constraints.
problem Online decision-making with constraints in episodic CMDPs.
method Optimistic planning using linear programming for PAC guarantee.
result Probably approximately correct (PAC) guarantee on episode count.
Algorithm improves online learning in adversarial bandits.
problem Online learning in adversarial multi-armed bandits with non-uniform best arm distribution.
method Online-within-online setup, inner and outer learners, leveraging non-uniform empirical distribution of best arms.
result Improves regret bounds for non-uniform best arm distributions.
New method improves robustness in partially observable domains by training against latent distribution shifts.
problem Challenges in robustness under latent distribution shift in partially observable reinforcement learning.
method Formalizes adversarial latent-initial-state POMDP, proves minimax principle, derives best-response inequalities.
result Reduces robustness gaps from 10.3 to 3.1 shots with targeted exposure to shifted latent distributions.
Paper proposes a method to optimize policies for diverse individuals using heterogeneous data.
problem Learning optimal policies for a heterogeneous population from pre-collected data.
method Individualized offline policy optimization framework for heterogeneous MDPs.
result The proposed P4L algorithm achieves a fast rate of average regret.
Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently,…
We introduce a new class of reinforcement learning methods referred to as {\em episodic multi-armed bandits} (eMAB). In eMAB the learner proceeds in {\em episodes}, each composed of several {\em steps}, in which it chooses an action and observes a feedback signal. Moreover, in each step, it can take a special action, c…
Paper proposes DAC-ML, a cognitive architecture that learns quickly from few episodes.
problem Sample inefficiency in AI learning action policies.
method Incorporates hippocampus-inspired sequential memory system into DAC theory of mind.
result DAC-ML rapidly converges to effective action policies maximizing reward.
In this paper we address the problem of discovering a small set of frequent serial episodes from sequential data so as to adequately characterize or summarize the data. We discuss an algorithm based on the Minimum Description Length (MDL) principle and the algorithm is a slight modification of an earlier method, called…
We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states.…
New rule reduces exploration regret to logarithmic, improving bad episode handling.
problem Improving exploration regret in average reward MDPs.
method Replacing Doubling Trick with Vanishing Multiplicative rule in EVI-based algorithms.
result Regret is logarithmic under the new rule, significantly better than linear.
Recurrent major mood episodes and subsyndromal mood instability cause substantial disability in patients with bipolar disorder. Early identification of mood episodes enabling timely mood stabilisation is an important clinical goal. Recent technological advances allow the prospective reporting of mood in real time enabl…
We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier. We propose an episodic memory model that performs sparse experience replay and local adaptation to mitigate catastrophic forgetting in this setup. Experiments on text classificatio…
New algorithms reduce dynamic regret in online MDPs with changing losses.
problem Online MDPs with adversarial loss changes and known transitions.
method Dynamic regret measure, novel ensemble algorithms for three models.
result Provably optimal dynamic regret bounds for episodic SSP, improved bounds for predictable environments.
Extends Thompson sampling for RL with fewer episodes.
problem Limited episodes in RL settings.
method Batch Bayesian optimization over episodes to learn action bias terms.
result Significantly outperforms standard Thompson sampling.
Single episode policy transfer in RL without access to rewards.
problem Performing near-optimally in a single attempt with unknown dynamics.
method Optimizes a probe and inference model to estimate latent variables for universal control policy.
result Significantly outperforms existing adaptive approaches in diverse domains.
Improved reinforcement learning for episodes with varying action sets.
problem Reinforcement learning with context-dependent action sets.
method Extends MVP algorithm to handle adversarial and stochastic contexts.
result Established minimax regret bounds of for adversarial contexts.
We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale. As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models agains…
Paper proposes new -regret measure for non-episodic RL.
problem Measuring performance in non-episodic RL environments.
method Introduces -regret as a new performance measure and derives bounds.
result Closed the gap between lower and upper bounds for -regret.
Recent advances in deep reinforcement learning algorithms have shown great potential and success for solving many challenging real-world problems, including Go game and robotic applications. Usually, these algorithms need a carefully designed reward function to guide training in each time step. However, in real world, …
New algorithm for average reward learning with bounded hitting time assumption.
problem Minimizing regret in average reward reinforcement learning with bounded hitting time.
method Optimistic Q-learning with a novel operator for bounded hitting time.
result Regret bound of for average reward learning.
This paper explores how environmental properties can simplify reinforcement learning in non-episodic settings.
problem Challenges in reinforcement learning with continuous interaction and sparse delayed rewards.
method Analysis of environment shaping and dynamism properties to simplify learning.
result Properties like environment shaping and dynamism can significantly ease learning in non-episodic, sparse reward settings.
Representing a dialog policy as a recurrent neural network (RNN) is attractive because it handles partial observability, infers a latent representation of state, and can be optimized with supervised learning (SL) or reinforcement learning (RL). For RL, a policy gradient approach is natural, but is sample inefficient. I…
New schemes improve lifelong learning by balancing old and new tasks.
problem Catastrophic forgetting in deep neural networks when learning multiple tasks.
method Unified optimization perspective of episodic memory based approaches, introducing MEGA-I and MEGA-II schemes.
result Significant improvement in lifelong learning benchmarks, reducing error by up to 18%.
MEMEC improves sample efficiency in reinforcement learning.
problem Lack of sample efficiency in reinforcement learning.
method Proposes MEMEC, a Boltzmann policy with state-dependent temperature for more principled exploration.
result MEMEC outperforms other methods on classic RL environments and Atari games.
Study finds stock prices rarely appreciate during capital inflows but often appreciate during normal flows.
problem Understanding stock price behavior during capital inflows and outflows.
method Identified capital flow episodes using threshold and k-means clustering; detected stock index changepoints using PELT method; combined results over identified capital flows.
result Stock prices rarely appreciate during capital inflows but often appreciate during normal flows.
In continual learning (CL), an agent learns from a stream of tasks leveraging prior experience to transfer knowledge to future tasks. It is an ideal framework to decrease the amount of supervision in the existing learning algorithms. But for a successful knowledge transfer, the learner needs to remember how to perform …
Many robotic applications require the agent to perform long-horizon tasks in partially observable environments. In such applications, decision making at any step can depend on observations received far in the past. Hence, being able to properly memorize and utilize the long-term history is crucial. In this work, we pro…
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.