Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.
problem Understanding and handling non-stationary environments in reinforcement learning.
method Demonstrated that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings.
result Transformers can approximate and learn strategies for non-stationary environments, matching or outperforming existing expert algorithms.
Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.
problem Challenges in non-stationary reinforcement learning, especially in updating value functions.
method Proved a worst-case complexity result for modifying reinforcement learning problems.
result Modifying reinforcement learning problems requires an amount of time almost as large as the number of states.
New algorithm tackles non-stationary reinforcement learning with general function approximation.
problem Understanding non-stationary MDPs with function approximation.
method Dynamic Bellman Eluder (DBE) dimension for complexity, sliding window mechanism, confidence set design.
result Upper bound on dynamic regret for proposed SW-OPEA algorithm.
KeRNS tackles non-stationary reinforcement learning in metric spaces.
problem Non-stationary reinforcement learning in metric spaces.
method KeRNS uses time-dependent kernels to model non-stationary Markov Decision Processes (MDPs).
result KeRNS achieves a regret bound that scales with the covering dimension and total variation of the MDP.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
Paper uses RL for market making, improving stability in non-stationary markets.
problem Optimizing market making strategies in non-stationary limit order book dynamics.
method Reinforcement Learning (Proximal-Policy Optimization) applied to a simulator.
result RL agent outperforms closed-form optimal solution in non-stationary markets.
High-dimensional always-changing environments constitute a hard challenge for current reinforcement learning techniques. Artificial agents, nowadays, are often trained off-line in very static and controlled conditions in simulation such that training observations can be thought as sampled i.i.d. from the entire observa…
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
Paper introduces a new method for improving reinforcement learning performance using transfer learning.
problem Improving reinforcement learning performance with limited sample sizes in dynamic decision-making scenarios.
method Developed a novel ``re-weighted targeting procedure'' and ``transfer deep Q ∗ Q^* Q ∗ -learning'' approach. result Demonstrated improved reinforcement learning performance through strategic sample construction.
DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
New RL method tackles dynamic MDPs with evolving rewards and states.
problem Dynamic MDPs with evolving rewards and states.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).
result Achieves dynamic regret bound for non-stationary MDPs.
LC-SAC tackles non-stationary dynamics in reinforcement learning.
problem Degradation of deep RL methods in non-stationary environments.
method LC-SAC uses latent context encoders and contrastive loss for dynamic information capture.
result LC-SAC outperforms SAC on environments with drastic dynamics changes.
New approach turns optimal stationary RL into non-stationary RL without prior knowledge.
problem Optimal RL in non-stationary environments without prior knowledge of non-stationarity.
method Black-box reduction of optimal stationary RL algorithms to non-stationary RL.
result Achieves optimal dynamic regret bounds in various RL settings.
NVMDP framework tackles non-stationary MDPs with varying discount rates.
problem Challenges in non-stationary environments and infinite-horizon formulations for reinforcement learning.
method Introduces NVMDP framework that accommodates non-stationarity and varying discount rates.
result NVMDPs provide a flexible mechanism to shape optimal policies without altering state or action spaces.
Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. Framework uses RL with dynamic embedding to outperform benchmarks in volatile markets.
problem Challenges in high-dimensional, non-stationary, and noisy market information.
method Dynamic embedding of market information using generative autoencoders and online meta-learning in a reinforcement learning framework.
result Framework outperforms common portfolio benchmarks and PTO approach during market stress.
New RL algorithm tackles non-stationary environments with flexible policy updates.
problem Non-stationary reinforcement learning with time-varying rewards and transition probabilities.
method Model-free policy-based algorithm NS-NAC with restart-based exploration and dynamic learning rates.
result Dynamic regret of i l d e O ( ∣ S ∣ 1 / 2 ∣ A ∣ 1 / 2 Δ T 1 / 6 T 5 / 6 ) ilde{\mathscr O}(|S|^{1/2}|A|^{1/2}Δ_T^{1/6}T^{5/6}) i l d e O ( ∣ S ∣ 1/2 ∣ A ∣ 1/2 Δ T 1/6 T 5/6 ) for both algorithms. Study improves self-driving safety in dynamic environments.
problem Safe self-driving in non-stationary urban settings.
method Neurosymbolic Meta-Reinforcement Lookahead Learning (NUMERLA).
result Self-driving agents can adapt safely in real-time.
New method extends supervised learning for non-stationary control problems.
problem Optimal control in non-stationary, reset-free environments.
method Prospective Learning with Control (PLuC) using Empirical Risk Minimization (ERM).
result ERM asymptotically achieves Bayes optimal policy in non-stationary environments.
PROPO tackles non-stationary MDPs with efficient policy optimization.
problem Non-stationary MDPs with varying reward and transition kernels.
method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.
result PROPO achieves near-optimal performance in non-stationary MDPs.
Optimistic algorithm reduces regret in non-stationary linear MDPs.
problem Efficient learning in non-stationary linear MDPs with evolving reward and transition.
method OPT-WLSVI, an optimistic model-free algorithm using exponential weights.
result Achieves a regret bound of O ~ ( d 5 / 4 H 2 Δ 1 / 4 K 3 / 4 ) \widetilde{\mathcal{O}}(d^{5/4}H^2 Δ^{1/4} K^{3/4}) O ( d 5/4 H 2 Δ 1/4 K 3/4 ) . New algorithms reduce dynamic regret in non-stationary RL environments.
problem Optimizing policies in environments that change over time.
method POWER and POWER++ algorithms for policy optimization with dynamic regret analysis.
result POWER++ improves dynamic regret by actively adapting to non-stationarity.
Reinforcement learning (RL) methods learn optimal decisions in the presence of a stationary environment. However, the stationary assumption on the environment is very restrictive. In many real world problems like traffic signal control, robotic applications, one often encounters situations with non-stationary environme…
We consider undiscounted reinforcement learning in Markov decision processes (MDPs) where both the reward functions and the state-transition probabilities may vary (gradually or abruptly) over time. For this problem setting, we propose an algorithm and provide performance guarantees for the regret evaluated against the…
Master algorithm fails to detect non-stationarity in practical settings.
problem Non-Stationary Reinforcement Learning without prior knowledge.
method Master algorithm tested under various conditions, including piecewise stationary multi-armed bandits.
result Master's non-stationarity detection is ineffective for practical horizons, leading to performance similar to random restarting.
New approach tackles non-stationary multi-agent games with black-box methods.
problem Challenges in learning equilibria in non-stationary multi-agent systems.
method Versatile black-box approach applicable to various games, including general-sum, potential, and Markov games.
result Achieves optimal regret bounds for non-stationary games, with or without knowledge of total variation.
New algorithm tackles non-stationary RL with near-optimal regret bounds.
problem Model-free reinforcement learning in non-stationary Markov decision processes.
method Proposed RestartQ-UCB algorithm with Freedman-type bonus terms.
result Achieves near-optimal dynamic regret bound in non-stationary RL.
Develops RL algorithm for lifelong non-stationary environments.
problem Challenges of reinforcement learning in environments with persistent change.
method Formalizes lifelong non-stationarity, uses latent variable models, and leverages online learning and probabilistic inference.
result Substantial improvement in performance over non-reasoning approaches in lifelong non-stationary environments.
Prognosticator improves performance in non-stationary MDPs.
problem Real-world applications often violate the stationary assumption in reinforcement learning.
method A policy gradient algorithm that maximizes a forecast of future performance.
result Prognosticator outperforms online adaptation techniques in non-stationary MDPs.
Develops RL algorithm for non-Markovian, non-stationary reward streams.
problem Maximizing rewards from non-Markovian, non-stationary reward streams.
method Uses causal DAG to construct Markov states, solves periodic MDP.
result Optimal state construction maximizes discounted rewards.
HTMRL uses HTM for RL, adapting faster to changing environments.
problem Adapting to non-stationary environments in RL.
method Strictly HTM-based RL algorithm.
result HTMRL adapts faster to changing environments in a 10-armed bandit.
Paper proposes a deep reinforcement learning model for forex trading that considers transaction costs.
problem Trading in forex markets with high transaction costs and non-stationary data.
method Deep reinforcement learning model considering transaction costs and online learning.
result Maximizes profit while keeping transaction costs low in non-stationary markets.
Adaptive Bayesian learning agent for non-stationary bandits.
problem Non-stationary rewards in reinforcement learning.
method Dynamic memory and statistical hypothesis testing.
result Adapts to changing rewards with minimal regret.
This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded ev…
Paper introduces novel Bandit algorithms for non-stationary environments in finance.
problem Non-stationary reward distributions in financial markets.
method Introduces Adaptive Discounted Thompson Sampling (ADTS) and Combinatorial Adaptive Discounted Thompson Sampling (CADTS) for non-stationary environments in portfolio optimization.
result Bandit Networks improve portfolio optimization performance by 20% compared to classical models.
Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off …
Unified approach for learning state representations from streaming data.
problem Learning reusable state representations from high-dimensional, non-stationary data.
method Unified mathematical formulation for learning latent relations, enabling flexible and principled shaping of latent space.
result Improved understanding and evaluation of existing unsupervised learning approaches.
Novel RL-based NPG improves multi-objective NAS efficiency and performance.
problem Discovering optimal neural architectures with multiple conflicting objectives.
method Non-stationary policy gradient with adaptive reward functions and shared model.
result Framework efficiently approximates full Pareto front and achieves superior performance.
A reinforcement learning method predicts medical outcomes from masked feature vectors.
problem Predicting medical outcomes from limited feature data.
method Reinforcement learning with iterative feature selection and a guesser network.
result The method outperforms baselines and is highly interpretable.
The study questions the necessity of accurate models in MBRL.
problem The difficulty of achieving and utilizing accurate models in complex or changing environments.
method Introduces a meta-learning algorithm focused on model usefulness rather than accuracy.
result Meta-learning models enable faster learning than accurate models built with domain-specific knowledge.
We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs) under temporal drifts, ie, both the reward and state transition distributions are allowed to evolve over time, as long as their respective total variations, quantified by suitable metrics, do not exceed certain variation budgets. …
FinFlowRL learns from experts to optimize financial control in changing markets.
problem Traditional finance control methods fail in real-world, non-stationary markets.
method Imitation-Reinforcement Learning framework that pretrains on expert strategies and finetunes in noise space.
result Consistently outperforms individually optimized experts across diverse market conditions.
When multiple agents learn in a decentralized manner, the environment appears non-stationary from the perspective of an individual agent due to the exploration and learning of the other agents. Recently proposed deep multi-agent reinforcement learning methods have tried to mitigate this non-stationarity by attempting t…
This paper presents a novel deep reinforcement learning-based resource allocation technique for the multi-agent environment presented by a cognitive radio network that coexists through underlay dynamic spectrum access (DSA) with a primary network. The resource allocation technique presented in this work is distributed,…
In this paper, the method UCB-RS, which resorts to recommendation system (RS) for enhancing the upper-confidence bound algorithm UCB, is presented. The proposed method is used for dealing with non-stationary and large-state spaces multi-armed bandit problems. The proposed method has been targeted to the problem of the …
A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…
Improved algorithm detects changes in RL environments with non-stationary MDPs.
problem Learning in non-stationary reinforcement learning environments.
method R-BOCPD-UCRL2 algorithm for MDPs with multinomial state transitions.
result Near-optimal theoretical guarantees in terms of false-alarm rate and detection delay.
Jelly Bean World tests never-ending learning in complex environments.
problem Current machine learning systems are specialized and limited in adaptability.
method Developed Jelly Bean World testbed for complex, non-stationary environments.
result Demonstrates the need for more adaptable machine learning systems.