Many interesting real world domains involve reinforcement learning (RL) in partially observable environments. Efficient learning in such domains is important, but existing sample complexity bounds for partially observable RL are at least exponential in the episode length. We give, to our knowledge, the first partially …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Variable Episode Lengths”
Study evaluates policies in partially observable environments without full model specification.
problem Evaluating policies in partially observable environments without full model specification.
method Developed non-parametric identification and recursive fitted-Q-evaluation algorithm.
result Established finite-sample error bounds for policy value estimation.
New algorithm reduces regret in stochastic shortest path problems.
problem Planning and control in environments with unknown dynamics and variable episode lengths.
method Developed an algorithm with a new regret bound of .
result Guaranteed a significant reduction in regret compared to previous methods.
We consider online learning in episodic loop-free Markov decision processes (MDPs), where the loss function can change arbitrarily between episodes, and the transition function is not known to the learner. We show regret bound, where is the number of episodes, is the state space, $A…
We study the dynamics of the linear and non-linear serial dependencies in financial time series in a rolling window framework. In particular, we focus on the detection of episodes of statistically significant two- and three-point correlations in the returns of several leading currency exchange rates that could offer so…
In this paper we address the problem of discovering a small set of frequent serial episodes from sequential data so as to adequately characterize or summarize the data. We discuss an algorithm based on the Minimum Description Length (MDL) principle and the algorithm is a slight modification of an earlier method, called…
Most provably-efficient learning algorithms introduce optimism about poorly-understood states and actions to encourage exploration. We study an alternative approach for efficient exploration, posterior sampling for reinforcement learning (PSRL). This algorithm proceeds in repeated episodes of known duration. At the sta…
New method learns near-optimal policies with polynomial samples in A and H.
problem Episodic latent MABs with partial observations are challenging.
method Experiment design and method-of-moments approach.
result Polynomial samples in A and H for near-optimal policy learning.
UCB-Advantage learns MDPs with regret.
problem Model-free reinforcement learning in finite-horizon MDPs.
method Reference-Advantage decomposition for low regret.
result Achieves regret, matching best known bounds.
PS4POMDPs algorithm simplifies online learning for episodic POMDPs with unknown models.
problem Learning in POMDPs is harder than in MDPs; online learning is especially challenging.
method Posterior Sampling-based reinforcement learning algorithm (PS4POMDPs)
result Bayesian regret scales as √number of episodes and is polynomial in other parameters.
Paper optimizes learning MDPs with adversarial rewards using linear models.
problem Learning MDPs with adversarial rewards and changing reward functions.
method Proposes POWERS, an optimistic policy optimization algorithm.
result Achieves nearly optimal regret bound of .
New Q-learning algorithms reduce regret in inventory control problems.
problem Efficiently learning optimal policies in inventory control problems with limited feedback.
method Proposed Elimination-Based Half-Q-Learning (HQL) and Full-Q-Learning (FQL) algorithms with theoretical regret bounds.
result HQL incurs regret, FQL incurs regret, independent of state and action space sizes.
SMRL uses score matching for efficient RL with exponential family models.
problem Efficient reinforcement learning with unknown transition models.
method Score matching for parameter estimation of exponential family models.
result Achieves online regret of .
The study identifies and analyzes different market regimes in equity markets using advanced signal processing techniques.
problem Understanding and quantifying the dynamics of different market regimes in equity markets.
method Data-driven Hilbert--Huang Transform for regime identification, Holo--Hilbert Spectral Analysis for profiling, and Variable-Length Markov Chains for return dynamics modeling.
result Developed markets normalize more effectively as stress subsides, while developing markets retain residual tail dependence and downside persistence.
New algorithms reduce regret in online MDPs by adapting to data and variance.
problem Adapting to both adversarial and stochastic environments in online MDPs.
method Develops algorithms based on global optimization and policy optimization, using optimistic follow-the-regularized-leader with log-barrier regularization.
result Achieves refined data-dependent and variance-dependent regret bounds.
New algorithm reduces regret and constraint violation in adversarial CMDP learning.
problem Online learning for episodic stochastically constrained Markov decision processes (CMDPs) with adversarial loss.
method Upper Confidence Primal-Dual Reinforcement Learning (UC-PDL) algorithm.
result Achieves upper bounds of both regret and constraint violation.
New algorithm learns optimal policies with just 1 episode, settling horizon-dependence in RL.
problem Understanding the sample complexity of reinforcement learning with horizon length.
method Developed an algorithm using only episodes to achieve PAC guarantee, leveraging connections between value functions in discounted and finite-horizon MDPs and novel perturbation analysis.
result Achieved the same PAC guarantee with only episodes of environment interactions, completely settling horizon-dependence in RL.
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…
New method trains GFlowNets from partial episodes to improve convergence and stability.
problem Improving convergence and stability of GFlowNets training.
method Introducing SubTB() for GFlowNet training from partial action subsequences.
result SubTB() accelerates GFlowNet convergence and enables training in longer action sequences.
Quantum UCB algorithm reduces reinforcement learning regret exponentially.
problem Episodic reinforcement learning with quantum state evolution.
method Upper Confidence Bound (UCB) quantum algorithm with quantum mean estimation.
result Exponential improvement in regret from $\Tilde{\mathcal{O}}(\sqrt{K})$ to $\Tilde{\mathcal{O}}(1)$.
New algorithm minimizes regret in sparse reinforcement learning.
problem Sparse reinforcement learning with unknown sparsity.
method Doubly robust approach combining feature vectors of all actions and novel analysis.
result Regret bound of .
New RL algorithm tackles adversarial RMAB with unknown transitions and bandit feedback.
problem Learning in episodic RMAB with unknown transition functions and adversarial rewards.
method Developed a novel RL algorithm with a biased reward estimator and an index policy.
result Achieved regret bound for adversarial RMAB.
Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a regret bound.
Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memorycs.NE
New memory allocation scheme improves image generation performance.
problem Improving episodic and semantic memory representation in neural networks.
method Developed a hierarchical latent variable model with differentiable, locally block allocated latent memory.
result Improved conditional likelihood values on various datasets.
Bootstrap method for Markov chains in reinforcement learning.
problem Distributional consistency in finite controlled Markov chains with unknown control policies.
method Model-based bootstrap with novel LLN and CLT for visitation counts and transition increments.
result Asymptotically valid confidence intervals for value and -functions in offline RL.
UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.
problem Improving Q-learning's bias and regret in reinforcement learning.
method UCBMQ combines Q-learning with an upper confidence bound and momentum term.
result UCBMQ guarantees a regret of with a linear second-order term in .
Current generation of memory-augmented neural networks has limited scalability as they cannot efficiently process data that are too large to fit in the external memory storage. One example of this is lifelong learning scenario where the model receives unlimited length of data stream as an input which contains vast majo…
Improved regret bound for MNL MDPs with variance-aware approach.
problem Optimal reinforcement learning for MNL MDPs with structured variance.
method Introducing a problem-dependent constant measuring average variance, proposing an algorithm with improved regret bound.
result Minimax optimal regret bound of for structured MDPs.
Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience …
Paper explores limits of imitation learning in MDPs, setting new suboptimality bounds.
problem Understanding the statistical limits of imitation learning in MDPs.
method Analyzes minimax statistical limits in two settings: pre-interaction and interaction.
result Establishes suboptimality bounds for imitation learning in MDPs, showing improvements with knowledge of transition.
Study on elastic curves with variable stiffness, derived from bending energy.
problem Modeling elastic wires with varying thickness.
method Derive Euler-Lagrange equations for curves with variable bending stiffness.
result Characterizations of elastic curves with variable stiffness.
New algorithm learns policies without explicit rewards for MDPs.
problem Learning policies for MDPs without explicit rewards.
method UCRL-RFE algorithm with linear function approximation.
result UCRL-RFE achieves optimal policy with minimal exploration samples.
Bayes-UCBVI tackles reinforcement learning with a new upper confidence bound method.
problem Optimizing exploration in reinforcement learning without bonuses.
method Bayes-UCBVI uses a quantile of a Q-value function posterior as an upper confidence bound.
result Proves a regret bound of order for tabular reinforcement learning.
In an episodic Markov Decision Process (MDP) problem, an online algorithm chooses from a set of actions in a sequence of trials, where is the episode length, in order to maximize the total payoff of the chosen actions. Q-learning, as the most popular model-free reinforcement learning (RL) algorithm, directly pa…
New algorithm learns POMDPs with known observation model efficiently.
problem Learning POMDPs with unknown transition model in average-reward setting.
method OAS estimation technique and OAS-UCRL algorithm balancing exploration-exploitation.
result Regret guarantee of order for OAS-UCRL algorithm.
Meta-learning framework improves short utterance speaker recognition.
problem Poor performance of existing models with short utterances.
method Prototypical Networks with support and query sets, enforcing classification against entire training set.
result Significant performance gains on VoxCeleb datasets.
Paper proposes a method to optimize policies for diverse individuals using heterogeneous data.
problem Learning optimal policies for a heterogeneous population from pre-collected data.
method Individualized offline policy optimization framework for heterogeneous MDPs.
result The proposed P4L algorithm achieves a fast rate of average regret.
SummerTime: Variable-length Time SeriesSummarization with Applications to PhysicalActivity Analysiscs.LG
SummerTime summarizes variable-length time series for machine learning applications.
problem Classical machine learning methods struggle with variable-length time series data.
method Summarizes time series into a fixed-length feature vector using Gaussian Mixture Models (GMM).
result Improves classification and regression performance in physical activity analysis.
Paper tackles regret bounds and exploration complexity for multi-objective reinforcement learning with picky preferences.
problem Formalizing multi-objective reinforcement learning with adversarial preferences.
method Model-based algorithm with nearly optimal regret bound and preference-free exploration.
result Achieves nearly minimax optimal regret bound and nearly optimal trajectory complexity.
A new RL approach optimizes reserve prices in multi-phase auctions, reducing revenue regret.
problem Optimizing reserve prices in multi-phase second-price auctions with noisy and potentially untruthful bidders.
method Combines RL techniques with buffer periods, a novel algorithm, and LSVI-UCB extension.
result Achieves optimal revenue regret under known and unknown noise conditions.
Study reveals how investor flows impact stock prices, especially during herding episodes.
problem Understanding how information transmits through prices and why it breaks down.
method Combining regularized deconvolution with Hawkes process analysis.
result Institutional price impact deteriorates sharply during herding episodes in small-cap stocks, while large-cap stocks maintain resilience.
Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscs.LG
Federated UCBVI reduces communication costs while minimizing regret in multi-agent settings.
problem Minimizing regret in federated learning with heterogeneous agents.
method Federated Upper Confidence Bound Value Iteration (Fed-UCBVI) algorithm.
result Regret bound scales as with small additional term due to heterogeneity.
Scaling laws found for reinforcement learning performance with model size and compute.
problem Challenges in extending generative modeling scaling laws to reinforcement learning.
method Introduced intrinsic performance as a monotonic function of mean episode return.
result Intrinsic performance scales as a power law in model size and environment interactions.
Study preference-based reinforcement learning in episodic kernel MDPs.
problem Learning from episodic human preferences in reinforcement learning.
method Developed preference-based value estimation and confidence sets for kernel-based MDPs.
result Proved high-probability regret bounds that converge to optimal policy value.
Algorithm tackles constrained reinforcement learning with concave-convex and knapsack constraints.
problem Constrained episodic reinforcement learning with concave rewards and convex constraints.
method Modular analysis with strong theoretical guarantees for concave-convex and knapsack settings.
result Significantly outperforms existing approaches in constrained episodic environments.
Algorithm improves multi-armed bandit performance by transferring reward samples.
problem Sequential multi-armed bandit problem with changing reward distributions.
method UCB algorithm with reward sample transfer.
result Significant improvement in cumulative regret over standard UCB.
BerlinUCB learns from episodic rewards in nonstationary contexts.
problem Online learning with episodic rewards in nonstationary environments.
method BerlinUCB integrates clustering for self-supervision.
result BerlinUCB outperforms standard contextual bandits in various scenarios.
New algorithm reduces dynamic regret for MDPs with unknown transition and adversarial rewards.
problem Episodic linear mixture MDPs with unknown transition and adversarial rewards.
method Combines occupancy-measure-based global optimization and policy-based variance-aware value-targeted regression.
result Achieves near-optimal dynamic regret of .