Efficient RL algorithms for linear function approximation with limited adaptivity constraints.
problem Limited adaptivity in reinforcement learning with linear function approximation.
method Proposed two efficient online RL algorithms for episodic linear Markov decision processes under batch learning and rare policy switch models.
result Achieved efficient regret bounds for both batch learning and rare policy switch models, with substantial reduction in adaptivity.
Two algorithms achieve optimal regret with limited adaptivity in multinomial logistic bandits.
problem Achieving optimal regret with limited adaptivity in multinomial logistic bandits.
method Presented two algorithms, B-MNL-CB and RS-MNL, for batched and rarely-switching paradigms.
result Achieved i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret with limited adaptivity. Excessively changing policies in many real world scenarios is difficult, unethical, or expensive. After all, doctor guidelines, tax codes, and price lists can only be reprinted so often. We may thus want to only change a policy when it is probable that the change is beneficial. In cases that a policy is a threshold on …
New RL algorithm achieves nearly optimal performance for linear MDPs.
problem Optimal reinforcement learning for episodic linear MDPs.
method Weighted linear regression with variance estimator and rare-switching policy.
result Achieves nearly minimax optimal regret i l d e O ( d H 3 K ) ilde O(d\sqrt{H^3K}) i l d e O ( d H 3 K ) . New actor-critic algorithm achieves optimal sample efficiency in RL.
problem Achieving ε ε ε -optimal policies with minimal samples in RL. method Integrates optimism, off-policy critic estimation, and rare-switching policy resets.
result Sample complexity of O ( d H 5 log ∣ A ∣ / ε 2 + d H 4 log ∣ F ∣ / ε 2 ) O(dH^5 \log|\mathcal{A}|/ε^2 + d H^4 \log|\mathcal{F}|/ ε^2) O ( d H 5 log ∣ A ∣/ ε 2 + d H 4 log ∣ F ∣/ ε 2 ) trajectories. We link disjoint longitudinal data for rare disease patients using latent representations and mixed-effects regression.
problem Analyzing treatment switches in rare diseases with limited data and changing measurement instruments.
method We embed item values into a shared latent space using variational autoencoders and apply mixed-effects regression to quantify treatment effects.
result Our approach allows for statistical inference and quantifies the impact of treatment switches in spinal muscular atrophy.
We develop a robust RL algorithm for off-dynamics environments with improved suboptimality bounds and computational efficiency.
problem Learning policies robust to uncertainties in transition dynamics between training and deployment environments.
method Distributionally robust Markov decision processes (DRMDPs) with a novel algorithm We-DRIVE-U.
result Improved suboptimality bound of O ~ ( d H ⋅ min { 1 / ρ , H } / K ) \widetilde{\mathcal{O}}\big({d H \cdot \min \{1/ρ, H\}/\sqrt{K} }\big) O ( d H ⋅ min { 1/ ρ , H } / K ) , near-optimal up to O ( H ) \mathcal{O}(\sqrt{H}) O ( H ) . Study tackles balancing policy switching costs in offline RL.
problem Balancing the cost of policy switching in offline RL.
method Optimal transport ideas and Net Actor-Critic algorithm.
result Demonstrated efficiency on multiple RL benchmarks.
New RL algorithm reduces policy switching cost to loglog(T) with similar regret.
problem Low policy switching cost in real-life RL applications.
method Stage-wise exploration and adaptive policy elimination.
result Regret of O ( H S A log log T ) O(HSA \log\log T) O ( H S A log log T ) with O ( H S A log log T ) O(HSA \log\log T) O ( H S A log log T ) switching cost. Study on adaptivity constraints in linear contextual bandits with optimal design.
problem Impact of adaptivity constraints on linear contextual bandits.
method Two models of limited adaptivity: batch learning and rare policy switches. Proposed distributional optimal design.
result Achieves minimax-optimal regret with optimal number of policy switches and batches.
A new policy switching technique improves offline RL performance.
problem Challenges in adapting off-policy algorithms to different datasets and tasks.
method Combines off-policy RL and BC, using epistemic uncertainty for policy switching.
result Outperforms individual algorithms and state-of-the-art methods on benchmarks.
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
The stochastic knapsack has been used as a model in wide ranging applications from dynamic resource allocation to admission control in telecommunication. In recent years, a variation of the model has become a basic tool in studying problems that arise in revenue management and dynamic/flexible pricing; and it is in thi…
The paper develops RL methods for optimal switching between multiple states.
problem Optimal switching between multiple states in continuous time.
method Entropy-regularized exploration, HJB equations, policy improvement, value function convergence.
result The RL algorithm converges to optimal policies as temperature parameter vanishes.
New algorithm reduces switching costs in RL beyond linear MDPs.
problem Costly policy switching in reinforcement learning.
method ELEANOR-LowSwitching algorithm for linear Bellman-complete MDPs.
result Achieves near-optimal regret with logarithmic switching cost.
New algorithm reduces RL complexity with low switching costs.
problem Exploration-exploitation dilemma in RL with complex models.
method Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation.
result Achieves minimax optimal regret of O ( d H K ) O(d\sqrt{HK}) O ( d H K ) and near-optimal policy switching cost. This paper first describes a class of uncertain stochastic control systems with Markovian switching, and derives an Itô-Liu formula for Markov-modulated processes. And we characterize an optimal control law, which satisfies the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching. Then, by using …
New RL algorithms reduce costs for single-agent and federated learning.
problem Minimizing costs in RL and federated RL settings.
method Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost algorithms.
result First algorithms to achieve low burn-in and logarithmic switching costs.
DG separates successes and failures by gating updates with advantage and surprisal.
problem Negative learning from surprising data in distributed reinforcement learning.
method DG gates each update with the product of advantage and surprisal, suppressing failures and preserving successes.
result DG outperforms other methods in various challenging reinforcement learning tasks.
Efficient algorithms for contextual slate bandits with limited adaptivity.
problem Contextual slate bandit problem with limited adaptivity.
method Proposed B-SlateGLinCB and RS-SlateGLinCB algorithms for batched and rarely-switching settings.
result Achieved regret bounds of O(Nd^(3/2)√T) and O(Nd√T) under diversity assumption.
Breaks down complex nonlinear dynamics into simpler components.
problem Control of nonlinear dynamical systems remains challenging.
method Inspired by hybrid switching systems, decomposes dynamics into simpler stochastic switching linear dynamical systems.
result Extracts hierarchies of Markovian and auto-regressive locally linear controllers from nonlinear experts.
New algorithm reduces decision switching in dynamic environments.
problem Online learning with memory and non-stationary environments.
method Dynamic policy regret, novel ensemble approach, meta-base decomposition.
result Proves optimal dynamic policy regret for memory length, non-stationarity, and time horizon.
Deep learning optimizes wireless band switching without measurement gaps.
problem Wireless networks waste data during band switching due to measurement gaps.
method Online-learning based classifier models exploiting spatial and spectral correlation.
result 30% improvement in mean effective rates compared to industry standard.
Algorithm learns to switch control among agents in a team.
problem Learning to switch control among reinforcement learning agents.
method 2-layer Markov decision process, upper confidence bounds, shared confidence bounds.
result Sublinear total regret with shared confidence bounds.
This work models market regimes using CTMSTOU and simulates trading policies.
problem Defining and understanding market regimes in finance.
method Discrete event time multi-agent market simulation with CTMSTOU model.
result Illustrates the importance of regime-awareness in trading policies.
New DR-IC estimator reduces bias and variance in OPE.
problem Estimating value of a target policy using logged data from a different policy.
method DR-IC estimator that combines parametric reward model and context-based switching rule.
result DR-IC estimator outperforms state-of-the-art OPE algorithms.
Optimizes consumption under regime-switching economic states with risk-sensitive preferences.
problem Optimizing consumption in an economy with uncertain states and random shocks.
method Risk-sensitive optimization of consumption-utility with a Markov chain model of economic states and i.i.d. random shocks.
result Existence of unique optimal policy and value function in stationary policies.
Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of…
Paper proves exponential lower bounds for policy iteration in MDPs.
problem Proving lower bounds for policy iteration in multi-action MDPs.
method Generalized previous results to k-action MDPs, constructed families of MDPs.
result Proved novel exponential lower bound of (3+k)2^(N/2-3) iterations.
PCGS-TF uses a Transformer to adaptively control expert switching in non-stationary environments.
problem Static regret is insufficient for strictly online prediction in non-stationary settings.
method Policy-Controlled Generalized Share (PCGS) with a Transformer as an update controller.
result PCGS-TF achieves the lowest dynamic regret in non-stationary families and expert pools.
In the hypothesis of rare loss events, the general expression of the policy value has been determined as a functional of the "expected frequency / loss severity" function and of the retention function. Exponential disutility has been chosen after mathematical characterization of some of its economical aspects, where fu…
New method improves reinforcement learning generalization.
problem Few environments lead to poor generalization in reinforcement learning.
method Integrates sequential structure into representation learning, using a policy similarity metric (PSM) and contrastive embeddings (PSEs).
result PSEs improve generalization across various benchmarks.
New algorithm learns optimal policies with minimal memory and time.
problem Learning optimal policies in discounted MDPs with short burn-in time.
method Variance reduction and adaptive policy switching.
result First regret-optimal model-free algorithm with low burn-in time.
New model for music streaming recommends songs based on past play history.
problem Nonstationary stochastic bandit model with delay-dependent rewards.
method Ranking policies approximating optimal policy with bounded regret.
result Algorithm with O ~ ( k T ) \widetilde{\mathcal{O}}\big(\!\sqrt{kT}\big) O ( k T ) regret and O ( k ln ln T ) \mathcal{O}\big(k\ln\ln T\big) O ( k ln ln T ) switches. A new method uses deep learning to predict rare events in complex systems.
problem Predicting rare and extreme events in non-equilibrium systems.
method A deep learning approach that minimizes the geometrical action.
result The method accurately predicts rare events in various complex systems.
H-ReIL learns to drive safely in near-accident scenarios.
problem Driving safely in high-risk near-accident situations.
method Hierarchical RL and IL approach.
result High-level policy switches between low-level policies for safe driving.
We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and …
Paper presents an efficient algorithm for linear MDP with low switching cost.
problem Large state space reinforcement learning problems with low switching cost.
method First algorithm for linear MDP with low switching cost, achieving near-optimal regret and switching cost.
result Regret bound of $\widetilde{O}\left(\sqrt{d^3H^4K}
ight)$ and near-optimal switching cost of $O\left(d H\log K
ight)$ .
Paper addresses uncertainty in model generalization under regime shifts.
problem Uncertainty in model generalization under regime changes.
method Proposes a framework to quantify and separate regime mismatch and sensitivity.
result Obtains exact decomposition and minimax lower bound for regime-aware models.
The study classifies policy announcements' impact on stock market volatility.
problem Evaluating the impact of Central Bank announcements on stock market volatility.
method Proposed a model-based classification method using Markov Switching dynamics and Multiplicative Error Model.
result Successful classification of 144 European Central Bank announcements on stock market volatility.
In this paper, we consider the optimal dividend problem for a company. We describe the surplus process of the company by a diffusion model with regime switching. The aim of the company is to choose a dividend policy to maximize the expected total discounted payments until ruin. In this article, we consider a hybrid div…
Study indifference pricing for insurance policies in a regime-switching market model.
problem Indifference pricing of pure endowment policies in a stochastic-factor model with different economic regimes.
method Stochastic control approach based on Hamilton-Jacobi-Bellman equation, Feynman-Kac formula, and sensitivity analysis.
result Characterization of indifference price as a solution to a linear PDE and a backward PDE.
This paper improves Q-learning bounds using reference-advantage decomposition.
problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.
method Develops a novel error decomposition framework to prove gap-dependent regret bounds.
result Establishes logarithmic gap-dependent regret bounds for Q-learning.
Optimal dividends strategy in a two-state regime-switching environment.
problem Maximizing profits from dividends until bankruptcy in a company with fluctuating cash surplus and regime changes in drift, volatility, and bankruptcy levels.
method Analyzes the optimal dividend payout strategy considering four factors: Brownian fluctuations in cash surplus, regime changes in drift, volatility, and bankruptcy levels.
result Rich structure of the optimal strategy, which can be either barrier-type or liquidation-barrier type, depending on model parameters.
We study the power of different types of adaptive (nonoblivious) adversaries in the setting of prediction with expert advice, under both full-information and bandit feedback. We measure the player's performance using a new notion of regret, also known as policy regret, which better captures the adversary's adaptiveness…
In a continuous time stochastic economy, this paper considers the problem of consumption and investment in a financial market in which the representative investor exhibits a change in the discount rate. The investment opportunities are a stock and a riskless account. The market coefficients and discount factor switches…
Adaptive framework improves NB accuracy by fusing two index categories.
problem Challenges in attribute weighted NB, especially fusion of two indexes.
method Proposes ATFNB framework using switching factor to fuse two index categories.
result ATFNB outperforms basic NB and state-of-the-art models.
This paper tackles near-optimal adversarial RL with switching costs, providing algorithms and matching lower bounds.
problem Adversarial RL with switching costs, where loss distribution can be non-stationary or adversarial.
method Developed novel switching-reduced algorithms with matching lower bounds for known and unknown transition functions.
result Achieved near-optimal performance in adversarial RL with switching costs, matching theoretical lower bounds.