Unified review of methods for inferring non-stationary process parameters.
problem Inferring parameters of non-stationary processes without a known model.
method Unified review and categorization of algorithms for Parameter Inference from a Non-stationary Unknown Process (PINUP).
result Simple statistical features can perform well on non-stationary systems, highlighting gaps in existing methods.
Non-stationarity appears in many online applications such as web search and advertising. In this paper, we study the online learning to rank problem in a non-stationary environment where user preferences change abruptly at an unknown moment in time. We consider the problem of identifying the K most attractive items and…
Algorithm adapts to non-stationary rewards without prior knowledge.
problem Optimizing decisions in non-stationary environments without prior knowledge of changes.
method Optimization-based algorithm that restarts when non-stationarity is detected.
result Achieves tighter dynamic regret bound and is nearly minimax optimal.
Study incentivizes exploration in non-stationary MAB with compensation.
problem Incentivized exploration for non-stationary stochastic bandits with biased feedback.
method Proposed algorithms for abruptly-changing and continuously-changing non-stationary environments.
result Achieves sublinear regret and compensation over time.
A new optimizer d-AmsGrad improves deep learning for robot learning in non-stationary problems.
problem Noise and outliers in real-world data make deep learning challenging for robot learning.
method Proposed an improved version of AmsGrad optimizer that slowly decays the maximum second momentum to adapt to non-stationary problems.
result The new optimizer outperformed baseline optimizers in robotics problems.
New algorithm for non-stationary bandits with slow drifts.
problem Minimizing dynamic regret in non-stationary bandits with slowly varying rewards.
method Extends Successive Elimination to non-stationary bandits with a novel gap profile characterization.
result First instance-dependent regret upper bound for slowly varying non-stationary bandits.
Paper tackles non-stationary kernelized bandits with near-optimal algorithm.
problem Minimizing regret in a time-varying reward function.
method Near-optimal algorithm with a novel restarting phased elimination with random permutation (R-PERP).
result Regret upper bound matches the lower bound, making the algorithm near-optimal.
Unified approach for non-stationary and clustered bandits.
problem Solving non-stationary and clustered bandits with overlapping solutions.
method Test of homogeneity for seamless integration of non-stationary and clustered bandits.
result Unified solution framework for change detection and cluster identification.
New algorithm tackles non-stationary reinforcement learning with general function approximation.
problem Understanding non-stationary MDPs with function approximation.
method Dynamic Bellman Eluder (DBE) dimension for complexity, sliding window mechanism, confidence set design.
result Upper bound on dynamic regret for proposed SW-OPEA algorithm.
We consider the problem of learning over non-stationary ranking streams. The rankings can be interpreted as the preferences of a population and the non-stationarity means that the distribution of preferences changes over time. Our goal is to learn, in an online manner, the current distribution of rankings. The bottlene…
Classic contextual bandit algorithms for linear models, such as LinUCB, assume that the reward distribution for an arm is modeled by a stationary linear regression. When the linear regression model is non-stationary over time, the regret of LinUCB can scale linearly with time. In this paper, we propose a novel multisca…
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
New algorithm tackles non-stationary delayed feedback in recommender systems.
problem Challenges in learning from delayed feedback in non-stationary environments.
method Developed a UCRL-based algorithm for non-stationary, delayed bandits with intermediate observations.
result Sublinear regret guarantees for the proposed algorithm in non-stationary delayed environments.
Study non-stationary bandits with resource constraints.
problem Maximize reward in a non-stationary environment with resource constraints.
method Propose new non-stationarity measure and use primal-dual analysis.
result Upper and lower bounds for non-stationary BwK problem.
Paper tackles non-stationary bandits with various examples.
problem Non-stationary stochastic bandit problem with specific cases.
method Proposes a single algorithm for multiple non-stationary bandit problems.
result Unified solution for four different bandit problems.
The paper extends NSGPs with L 1 L^1 L 1 -regularization for sparsity and solves the resulting R-NSGP regression problem.
problem Sparsity in non-stationary temporal data.
method Developed an ADMM-based method for solving the regularized NSGP regression problem.
result The proposed methods induce sparsity in the parameters of NSGPs.
Reinforcement learning (RL) methods learn optimal decisions in the presence of a stationary environment. However, the stationary assumption on the environment is very restrictive. In many real world problems like traffic signal control, robotic applications, one often encounters situations with non-stationary environme…
Algorithm reduces regret in non-stationary bandits and meta-learning with optimal arms.
problem Sequential decision-making with changing task boundaries and optimal arms.
method Reduction to bandit submodular maximization, meta-learning algorithms.
result Regret bounds for both non-stationary and bandit meta-learning problems.
Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.
problem Challenges in non-stationary reinforcement learning, especially in updating value functions.
method Proved a worst-case complexity result for modifying reinforcement learning problems.
result Modifying reinforcement learning problems requires an amount of time almost as large as the number of states.
Improved algorithm for adaptive dueling bandits with near-optimal regret bound.
problem Non-stationary dueling bandits with unknown number of preference changes.
method Elimination-based rescheduling algorithm for adaptive dynamic regret.
result Near-optimal i l d e O ( S e x t t t C W T ) ilde{O}(\sqrt{S^{ exttt{CW}} T}) i l d e O ( S e x ttt C W T ) dynamic regret bound. Algorithm minimizes regret in non-stationary dueling bandits with unknown parameters.
problem Minimizing regret in dueling bandits with time-varying preferences.
method Proposes Beat the Winner Reset algorithm and meta-algorithms DETECT and Monitored Dueling Bandits.
result Proves bounds on expected weak and strong regret for non-stationary dueling bandits.
New RL algorithm tackles non-stationary environments with flexible policy updates.
problem Non-stationary reinforcement learning with time-varying rewards and transition probabilities.
method Model-free policy-based algorithm NS-NAC with restart-based exploration and dynamic learning rates.
result Dynamic regret of i l d e O ( ∣ S ∣ 1 / 2 ∣ A ∣ 1 / 2 Δ T 1 / 6 T 5 / 6 ) ilde{\mathscr O}(|S|^{1/2}|A|^{1/2}Δ_T^{1/6}T^{5/6}) i l d e O ( ∣ S ∣ 1/2 ∣ A ∣ 1/2 Δ T 1/6 T 5/6 ) for both algorithms. DAL enhances black-box bandit algorithms for non-stationary environments.
problem Non-stationary environments in bandit problems.
method DAL combines any stationary bandit algorithm with a change detector.
result DAL consistently outperforms state-of-the-art methods in various non-stationary scenarios.
DS-TS adapts to abrupt and smooth changes in bandit problems.
problem Non-stationary multi-armed bandit problems with abrupt and smooth changes.
method Discounted Thompson Sampling with Gaussian priors.
result Achieves nearly optimal regret bound for both abrupt and smooth changes.
Learning in a non-stationary environment is an inevitable problem when applying machine learning algorithm to real world environment. Learning new tasks without forgetting the previous knowledge is a challenge issue in machine learning. We propose a Kalman Filter based modifier to maintain the performance of Neural Net…
New method reduces dynamic regret for non-stationary bandits.
problem Non-stationary stochastic multi-armed bandit problem with changing optimal arm.
method Proposes a method achieving near-optimal dynamic regret without prior knowledge of changes.
result Achieves O ~ ( K N ( S + 1 ) ) \widetilde O(\sqrt{K N(S+1)}) O ( K N ( S + 1 ) ) dynamic regret. New definition resolves ambiguity in non-stationary bandit classification.
problem Ambiguity in classifying non-stationary bandits using existing definitions.
method Introducing a formal definition that resolves ambiguity and provides a unified approach.
result Unified approach applicable to both Bayesian and frequentist formulations, resolves classification issues.
Model separates overall uncertainty into aleatoric and epistemic components for active learning.
problem Active learning with uncertainty quantification.
method Non-stationary Heteroscedastic Gaussian process model.
result Model separates overall uncertainty into aleatoric and epistemic components.
We revisit the classical Douglas-Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the stepsizes, we aim at developing an adaptive stepsize rule. To that end, we take a closer look at a linear case of the problem a…
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
Algorithm minimizes control regret for non-stationary LQR systems.
problem Control of non-stationary LQR systems with unknown dynamics.
method Adaptive non-stationarity detection and OLS estimator with small bias.
result Achieves optimal dynamic regret of $ ilde{\mathcal{O}}\left(V_T^{2/5}T^{3/5}
ight)$ .
New TS algorithms improve performance in non-stationary multi-armed bandit problems.
problem Sequential decision-making with evolving action rewards.
method Sliding-window Thompson sampling approaches with different priors.
result Unified regret upper bound for arbitrary non-stationary MABs.
Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. In this paper, we study the non-stationary online second price auction problem. We assume that the seller is selling the same type of items in T T T rounds by the second price auction, and she can set the reserve price in each round. In each round, the bidders draw their private values from a joint distribution unknown t…
The application of Gaussian processes (GPs) to large data sets is limited due to heavy memory and computational requirements. A variety of methods has been proposed to enable scalability, one of which is to exploit structure in the kernel matrix. Previous methods, however, cannot easily deal with non-stationary process…
New method extends supervised learning for non-stationary control problems.
problem Optimal control in non-stationary, reset-free environments.
method Prospective Learning with Control (PLuC) using Empirical Risk Minimization (ERM).
result ERM asymptotically achieves Bayes optimal policy in non-stationary environments.
Optimistic algorithm reduces regret in non-stationary linear MDPs.
problem Efficient learning in non-stationary linear MDPs with evolving reward and transition.
method OPT-WLSVI, an optimistic model-free algorithm using exponential weights.
result Achieves a regret bound of O ~ ( d 5 / 4 H 2 Δ 1 / 4 K 3 / 4 ) \widetilde{\mathcal{O}}(d^{5/4}H^2 Δ^{1/4} K^{3/4}) O ( d 5/4 H 2 Δ 1/4 K 3/4 ) . Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.
problem Understanding and handling non-stationary environments in reinforcement learning.
method Demonstrated that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings.
result Transformers can approximate and learn strategies for non-stationary environments, matching or outperforming existing expert algorithms.
Incremental learning from non-stationary data poses special challenges to the field of machine learning. Although new algorithms have been developed for this, assessment of results and comparison of behaviors are still open problems, mainly because evaluation metrics, adapted from more traditional tasks, can be ineffec…
Proposes a method to forecast non-stationary time series.
problem Challenges of non-stationary conditional distributions in deep learning.
method Bayesian dynamic model + deep conditional distribution model.
result Adapts to non-stationary time series better than state-of-the-art solutions.
New definitions of ESP for quantum reservoir computing handle non-stationary systems.
problem Traditional ESP does not apply to non-stationary systems.
method Introduce two new categories of ESP: non-stationary ESP and subset/subspace ESP.
result Demonstrates correspondence between non-stationary ESP and QRC with NARMA tasks.
Study combines VICReg and TNC for better encoding of non-stationary seismic signals.
problem Ineffective self-supervised learning on non-stationary time series.
method Combines VICReg and Temporal Neighborhood Coding (TNC).
result Effective for self-supervised learning on non-stationary seismic signals.
We propose non-stationary spectral kernels for Gaussian process regression. We propose to model the spectral density of a non-stationary kernel function as a mixture of input-dependent Gaussian process frequency density surfaces. We solve the generalised Fourier transform with such a model, and present a family of non-…
DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
Study MNL-Bandit in non-stationary settings with optimal regret bound.
problem Optimizing decisions in a non-stationary environment for multi-armed bandit problems.
method Develops an algorithm with worst-case expected regret bound and introduces new techniques to handle non-stationarity.
result Optimal regret bound proven for the MNL-Bandit problem in non-stationary environments.
New method optimizes policies in non-stationary environments.
problem Optimizing policies in non-stationary, context-dependent environments.
method Two-phase approach: offline learning and online adaptation.
result Our method outperforms existing approaches in both synthetic and real-world datasets.
New GLB algorithm reduces regret in non-stationary settings.
problem Non-stationary environments in GLBs.
method Adapted projection step for tracking parameter-drift.
result Regret bound of i l d e O ( B T 1 / 3 T 2 / 3 ) ilde{\mathcal{O}}(B_T^{1/3}T^{2/3}) i l d e O ( B T 1/3 T 2/3 ) under geometric action set.