On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Non-stationary MDPs”

Non-stationary Reinforcement Learning under General Function Approximationcs.LG

New algorithm tackles non-stationary reinforcement learning with general function approximation.

problem Understanding non-stationary MDPs with function approximation.

method Dynamic Bellman Eluder (DBE) dimension for complexity, sliding window mechanism, confidence set design.

result Upper bound on dynamic regret for proposed SW-OPEA algorithm.

non-stationary reinforcement learning dynamic Bellman Eluder dimension confidence set sliding window dynamic regret

2023-06-01abs ↗pdf ↗

Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcs.LG

Paper establishes new lower bounds for MDPs with changing transition kernels.

problem Minimizing sample complexity and regret in non-stationary MDPs.

method Developed novel lower bounds and constructed hard MDPs.

result Proved

Ω((H^3SA/ε^2)\log(1/δ))

sample complexity lower bound.

Reinforcement Learning MDP Non-stationary Lower Bounds PAC

2020-10-07abs ↗pdf ↗

MetaCURL: Non-stationary Concave Utility Reinforcement Learningcs.LG

MetaCURL tackles non-stationary MDPs with optimal dynamic regret.

problem Online learning in non-stationary Markov decision processes.

method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.

result Achieves optimal dynamic regret without prior knowledge of MDP changes.

Reinforcement Learning Non-stationary MDPs Meta-algorithm Optimal Regret Sleeping Expert

2024-05-30abs ↗pdf ↗

DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guaranteescs.LG

DARLING tackles non-stationary RL with guarantees, improving dynamic regret.

problem Non-stationary reinforcement learning in unknown change points.

method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.

result DARLING matches minimax lower bounds in tabular and linear MDPs.

reinforcement learning non-stationary dynamic regret tabular MDPs linear MDPs

2026-04-17abs ↗pdf ↗

A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescs.LG

KeRNS tackles non-stationary reinforcement learning in metric spaces.

problem Non-stationary reinforcement learning in metric spaces.

method KeRNS uses time-dependent kernels to model non-stationary Markov Decision Processes (MDPs).

result KeRNS achieves a regret bound that scales with the covering dimension and total variation of the MDP.

reinforcement learning non-stationary metric spaces kernel-based episodic

2020-07-09abs ↗pdf ↗

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPscs.LG

PROPO tackles non-stationary MDPs with efficient policy optimization.

problem Non-stationary MDPs with varying reward and transition kernels.

method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.

result PROPO achieves near-optimal performance in non-stationary MDPs.

reinforcement learning non-stationary MDPs optimistic policy optimization periodic restart linear function approximation

2021-10-18abs ↗pdf ↗

Efficient Learning in Non-Stationary Linear Markov Decision Processescs.LG

Optimistic algorithm reduces regret in non-stationary linear MDPs.

problem Efficient learning in non-stationary linear MDPs with evolving reward and transition.

method OPT-WLSVI, an optimistic model-free algorithm using exponential weights.

result Achieves a regret bound of

\widetilde{\mathcal{O}}(d^{5/4}H^2 Δ^{1/4} K^{3/4})

reinforcement learning non-stationary MDPs optimistic algorithms weighted least squares exponential weights

2020-10-24abs ↗pdf ↗

Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learningcs.LG

NVMDP framework tackles non-stationary MDPs with varying discount rates.

problem Challenges in non-stationary environments and infinite-horizon formulations for reinforcement learning.

method Introduces NVMDP framework that accommodates non-stationarity and varying discount rates.

result NVMDPs provide a flexible mechanism to shape optimal policies without altering state or action spaces.

Reinforcement Learning Markov Decision Processes Non-stationary Varying Discounting Optimal Policy

2025-11-17abs ↗pdf ↗

Revisiting Weighted Strategy for Non-stationary Parametric Bandits and MDPscs.LG

This paper refines the weighted strategy for non-stationary parametric bandits and MDPs, improving regret bounds.

problem Non-stationary environments with gradual drifting patterns.

method Refined analysis framework for the weighted strategy, leading to simpler and more efficient algorithms.

result Improved regret bounds for linear bandits, generalized linear bandits, and self-concordant bandits.

non-stationary parametric bandits weighted strategy regret analysis linear bandits

2026-01-03abs ↗pdf ↗

Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcs.LG

New approach turns optimal stationary RL into non-stationary RL without prior knowledge.

problem Optimal RL in non-stationary environments without prior knowledge of non-stationarity.

method Black-box reduction of optimal stationary RL algorithms to non-stationary RL.

result Achieves optimal dynamic regret bounds in various RL settings.

Reinforcement Learning Non-stationary Optimal Regret Black-box Reduction Bandits

2021-02-10abs ↗pdf ↗

Restarted Bayesian Online Change-point Detection for Non-Stationary Markov Decision Processescs.LG

Improved algorithm detects changes in RL environments with non-stationary MDPs.

problem Learning in non-stationary reinforcement learning environments.

method R-BOCPD-UCRL2 algorithm for MDPs with multinomial state transitions.

result Near-optimal theoretical guarantees in terms of false-alarm rate and detection delay.

Bayesian Change-point detection Reinforcement learning Non-stationary Markov decision process

2023-04-01abs ↗pdf ↗

Optimizing for the Future in Non-Stationary MDPscs.LG

Prognosticator improves performance in non-stationary MDPs.

problem Real-world applications often violate the stationary assumption in reinforcement learning.

method A policy gradient algorithm that maximizes a forecast of future performance.

result Prognosticator outperforms online adaptation techniques in non-stationary MDPs.

reinforcement learning non-stationary MDPs policy gradient forecast performance maximization

2020-05-17abs ↗pdf ↗

Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcs.LG

New RL method tackles dynamic MDPs with evolving rewards and states.

problem Dynamic MDPs with evolving rewards and states.

method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).

result Achieves dynamic regret bound for non-stationary MDPs.

Reinforcement Learning Markov Decision Processes Non-Stationarity Dynamic Regret Confidence Widening

2020-06-24abs ↗pdf ↗

Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncs.LG

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded ev…

2019-04-22abs ↗pdf ↗

Harnessing Causality in Reinforcement Learning With Bagged Decision Timescs.LG

Develops RL algorithm for non-Markovian, non-stationary reward streams.

problem Maximizing rewards from non-Markovian, non-stationary reward streams.

method Uses causal DAG to construct Markov states, solves periodic MDP.

result Optimal state construction maximizes discounted rewards.

Reinforcement Learning Causality Non-Markovian Non-Stationary Bayesian

2024-10-18abs ↗pdf ↗

Efficient Policy Learning for Non-Stationary MDPs under Adversarial Manipulationcs.LG

A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…

2019-07-22abs ↗pdf ↗

Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPscs.LG

New algorithm reduces dynamic regret for MDPs with unknown transition and adversarial rewards.

problem Episodic linear mixture MDPs with unknown transition and adversarial rewards.

method Combines occupancy-measure-based global optimization and policy-based variance-aware value-targeted regression.

result Achieves near-optimal dynamic regret of

\widetilde{\mathcal{O}}(d \sqrt{H^3 K} + \sqrt{HK(H + \bar{P}_K)})

dynamic regret linear mixture MDPs unknown transition adversarial rewards occupancy-measure-based

2024-11-05abs ↗pdf ↗

A View of the Certainty-Equivalence Method for PAC RL as an Application of the Trajectory Tree Methodcs.LG

This paper shows CEM is a special case of TTM, leading to new proofs and improved sample complexity bounds.

problem Improving sample complexity for reinforcement learning algorithms.

method Viewing CEM as an application of TTM, deriving new proofs and bounds.

result Improved sample complexity bounds for CEM under various conditions.

reinforcement learning certainty-equivalence method trajectory tree method sample complexity minimax-optimality

2025-01-05abs ↗pdf ↗

Variational Regret Bounds for Reinforcement Learningcs.LG

We consider undiscounted reinforcement learning in Markov decision processes (MDPs) where both the reward functions and the state-transition probabilities may vary (gradually or abruptly) over time. For this problem setting, we propose an algorithm and provide performance guarantees for the regret evaluated against the…

2019-05-14abs ↗pdf ↗

Stochastic Variance-Reduced Policy Gradientcs.LG

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation…

2018-06-14abs ↗pdf ↗

Dynamic Regret of Policy Optimization in Non-stationary Environmentscs.LG

New algorithms reduce dynamic regret in non-stationary RL environments.

problem Optimizing policies in environments that change over time.

method POWER and POWER++ algorithms for policy optimization with dynamic regret analysis.

result POWER++ improves dynamic regret by actively adapting to non-stationarity.

Reinforcement Learning Policy Optimization Dynamic Regret Non-stationary Environments Adaptation

2020-06-30abs ↗pdf ↗

Non-Stationary Dividend-Price Ratiosq-fin.PM

Dividend yields have been widely used in previous research to relate stock market valuations to cash flow fundamentals. However, this approach relies on the assumption that dividend yields are stationary. Due to the failure to reject the hypothesis of a unit root in the classical dividend-price ratio for the US stock m…

2019-02-16abs ↗pdf ↗

Non-Stationary Reinforcement Learning: The Blessing of (More) Optimismcs.LG

We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs) under temporal drifts, ie, both the reward and state transition distributions are allowed to evolve over time, as long as their respective total variations, quantified by suitable metrics, do not exceed certain variation budgets. …

2019-06-07abs ↗pdf ↗

Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncs.LG

New algorithm reduces offline RL data requirements significantly.

problem Optimizing policies using only historical data in reinforcement learning.

method Off-Policy Double Variance Reduction (OPDVR) algorithm.

result OPDVR achieves optimal sample complexity with

O(H^2/d_mε^2)

episodes.

reinforcement learning offline RL variance reduction policy optimization sample complexity

2021-02-02abs ↗pdf ↗

Relax but stay in control: from value to algorithms for online Markov decision processescs.LG

Online learning algorithms are designed to perform in non-stationary environments, but generally there is no notion of a dynamic state to model constraints on current and future actions as a function of past actions. State-based models are common in stochastic control settings, but commonly used frameworks such as Mark…

2013-10-28abs ↗pdf ↗

A Definition of Non-Stationary Banditscs.LG

New definition resolves ambiguity in non-stationary bandit classification.

problem Ambiguity in classifying non-stationary bandits using existing definitions.

method Introducing a formal definition that resolves ambiguity and provides a unified approach.

result Unified approach applicable to both Bayesian and frequentist formulations, resolves classification issues.

non-stationary bandits regret dynamic regret weak regret agent performance

2023-02-23abs ↗pdf ↗

Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPscs.LG

Adaptive reduction scheme approximates optimal policy in regularized MDPs.

problem Finding near optimal policy in regularized MDPs with biased solutions.

method Adaptive reduction of regularization parameter λ to approximate optimal policy.

result Iteration complexity reduced for obtaining ε-optimal policy.

MDPs Regularization Policy approximation Adaptive reduction Optimization

2020-10-31abs ↗pdf ↗

Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learningstat.ML

We study Exo-MDPs to reduce sample complexity in reinforcement learning.

problem Reducing sample complexity in reinforcement learning for structured MDPs.

method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.

result Proved

O(H^{3/2}d\sqrt{K})

regret bound for Exo-MDPs, matching lower bounds.

Reinforcement Learning Markov Decision Processes Exogenous States Sample-Efficient Inventory Control

2024-09-22abs ↗pdf ↗

Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcs.LG

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…

2017-09-19abs ↗pdf ↗

Non-Stationary Functional Bilevel Optimizationstat.ML

SmoothFBO tackles non-stationary functional bilevel optimization.

problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.

method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.

result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.

functional bilevel optimization non-stationary optimization hyperparameter optimization reinforcement learning sublinear regret

2026-01-21abs ↗pdf ↗

Solving Continual Combinatorial Selection via Deep Reinforcement Learningcs.LG

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …

2019-09-09abs ↗pdf ↗

Multiscale Non-stationary Stochastic Banditsstat.ML

Proposes a new algorithm for non-stationary bandits.

problem Non-stationary reward distributions in contextual bandits.

method Multiscale changepoint detection for adaptive learning.

result Regret bound analysis and superior performance in experiments.

bandits non-stationary multiscale changepoint adaptive

2020-02-13abs ↗pdf ↗

Optimality Guarantees for Particle Belief Approximation of POMDPscs.AI

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.

method Bounding particle filtering error and adapting MDP algorithms to POMDPs.

result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.

POMDPs Particle Belief Approximation MDPs Error Bounds Continuous Spaces

2022-10-10abs ↗pdf ↗

Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learningstat.ML

Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.

problem Understanding and handling non-stationary environments in reinforcement learning.

method Demonstrated that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings.

result Transformers can approximate and learn strategies for non-stationary environments, matching or outperforming existing expert algorithms.

Transformers Reinforcement Learning Non-Stationary Dynamic Regret In-Context Learning

2025-08-22abs ↗pdf ↗

Incentivized Exploration of Non-Stationary Stochastic Banditscs.LG

Study incentivizes exploration in non-stationary MAB with compensation.

problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.

method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.

result Achieves sublinear regret and compensation over time.

bandits incentivized exploration non-stationary compensation regret

2024-03-16abs ↗pdf ↗

DynaConF: Dynamic Forecasting of Non-Stationary Time Seriescs.LG

Proposes a method to forecast non-stationary time series.

problem Challenges of non-stationary conditional distributions in deep learning.

method Bayesian dynamic model + deep conditional distribution model.

result Adapts to non-stationary time series better than state-of-the-art solutions.

time series forecasting non-stationary deep learning conditional distribution Bayesian model

2022-09-17abs ↗pdf ↗

Extending echo state property for quantum reservoir computingquant-ph

New definitions of ESP for quantum reservoir computing handle non-stationary systems.

problem Traditional ESP does not apply to non-stationary systems.

method Introduce two new categories of ESP: non-stationary ESP and subset/subspace ESP.

result Demonstrates correspondence between non-stationary ESP and QRC with NARMA tasks.

quantum reservoir computing echo state property non-stationary systems subspace ESP NARMA tasks

2024-03-05abs ↗pdf ↗

Scaling Up Robust MDPs by Reinforcement Learningcs.LG

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…

2013-06-26abs ↗pdf ↗

VNIbCReg: VICReg with Neighboring-Invariance and better-Covariance Evaluated on Non-stationary Seismic Signal Time Seriescs.LG

Study combines VICReg and TNC for better encoding of non-stationary seismic signals.

problem Ineffective self-supervised learning on non-stationary time series.

method Combines VICReg and Temporal Neighborhood Coding (TNC).

result Effective for self-supervised learning on non-stationary seismic signals.

self-supervised learning non-stationary time series VICReg TNC seismic signal

2022-04-06abs ↗pdf ↗

Non-Stationary Spectral Kernelsstat.ML

We propose non-stationary spectral kernels for Gaussian process regression. We propose to model the spectral density of a non-stationary kernel function as a mixture of input-dependent Gaussian process frequency density surfaces. We solve the generalised Fourier transform with such a model, and present a family of non-…

2017-05-24abs ↗pdf ↗

Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPscs.LG

Optimizes learning policies in MDPs with weakly communicating structure.

problem Learning optimal policies in weakly communicating MDPs with generative model.

method Span-based approach, reducing to discounted MDPs for analysis.

result First minimax optimal sample complexity bound for weakly communicating MDPs.

Markov decision process sample complexity weakly communicating average-reward transient time

2024-03-18abs ↗pdf ↗

An Optimization-based Algorithm for Non-stationary Kernel Bandits without Prior Knowledgestat.ML

Algorithm adapts to non-stationary rewards without prior knowledge.

problem Optimizing decisions in non-stationary environments without prior knowledge of changes.

method Optimization-based algorithm that restarts when non-stationarity is detected.

result Achieves tighter dynamic regret bound and is nearly minimax optimal.

non-stationary bandits optimization dynamic regret kernel methods neural networks

2022-05-29abs ↗pdf ↗

Robust Markov Decision Processes without Model Estimationstat.ML

New method removes oracle and reduces memory usage for robust MDPs.

problem Applying robust MDPs in practice due to model estimation and oracle requirements.

method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.

result Sample-efficient algorithm with lower storage requirement and no oracle.

Markov Decision Processes Robustness Stochastic Gradient Model-Free Oracle

2023-02-02abs ↗pdf ↗

DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscs.LG

DeepAveragers solves offline RL by solving derived MDPs from static data.

problem Offline reinforcement learning with limited data.

method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.

result The approach can lower-bound performance and scale to complex offline RL problems.

offline reinforcement learning non-parametric MDPs deep representations costs performance bounds

2020-10-18abs ↗pdf ↗

Online RL in Linearly

q^π

-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignorecs.LG

New RL method learns to skip states in linearly $q^π$ -realizable MDPs, simplifying to linear MDPs.

problem Online RL in episodic MDPs with linearly

q^π

-realizable action-values.

method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.

result First polynomial-sample-complexity online RL algorithm for linearly

q^π

-realizable MDPs.

online RL Markov decision processes linearly $q^π$-realizable skip states misspecified

2023-10-11abs ↗pdf ↗

Exact Simulation of Non-stationary Reflected Brownian Motionmath.PR

This paper develops the first method for the exact simulation of reflected Brownian motion (RBM) with non-stationary drift and infinitesimal variance. The running time of generating exact samples of non-stationary RBM at any time $t$ is uniformly bounded by $\mathcal{O}(1/\barγ^2)$ where $\barγ$ is the average drift of…

2013-12-23abs ↗pdf ↗

Span-Based Optimal Sample Complexity for Average Reward MDPscs.LG

Optimizes learning policies in average-reward MDPs with improved sample complexity.

problem Learning optimal policies in average-reward MDPs with limited samples.

method Reduces to discounted MDPs and uses improved bounds for variance parameters.

result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))

Markov decision process average reward sample complexity discounted MDP optimal policy

2023-11-22abs ↗pdf ↗

Offline Dynamic Inventory and Pricing Strategy: Addressing Censored and Dependent Demandstat.ML

Study optimal pricing and inventory control in dynamic settings with censored demand.

problem Optimal pricing and inventory control in dynamic settings with censored demand.

method Approximate optimal policy via high-order MDP, propose novel algorithms for solving Bellman equations.

result Established finite-sample regret bounds and demonstrated efficacy through numerical experiments.

offline dynamic inventory pricing strategy censored demand Markov decision process survival analysis

2025-04-14abs ↗pdf ↗