The paper tackles non-stationary MAB with periodic rewards.
problem Non-stationary mean rewards over time in a business context.
method Combines Fourier analysis with confidence-bound learning to estimate periods and minimize regret.
result Proposes a near-optimal policy with a regret bound of O ( T ∑ k = 1 K T k ) O(\sqrt{T\sum_{k=1}^K T_k}) O ( T ∑ k = 1 K T k ) . New policy for non-stationary multi-armed bandits with recovering rewards.
problem Non-stationary multi-armed bandits with recovering rewards.
method Purely Periodic Policies integrating upper confidence bound procedure.
result Approximately O ~ ( N T ) \widetilde{\mathcal O}(N\sqrt{T}) O ( N T ) regret against the offline benchmark. Adaptive MAB algorithms handle composite, anonymous feedback without reward interval knowledge.
problem Multi-armed bandit with composite and anonymous feedback, especially without reward interval size knowledge.
method Proposed adaptive algorithms for stochastic and adversarial cases, without reward interval knowledge.
result First algorithm for adversarial case handling non-oblivious adversary and unknown reward interval size.
In retail, there are predictable yet dramatic time-dependent patterns in customer behavior, such as periodic changes in the number of visitors, or increases in customers just before major holidays. The current paradigm of multi-armed bandit analysis does not take these known patterns into account. This means that for a…
Algorithm tackles clustered contextual bandits with resource constraints.
problem Maximizing reward while respecting resource limits in unknown cluster contexts.
method Combines econometrics and bandit constraints techniques for sublinear regret.
result Achieves sublinear regret without needing all arm information.
Develops RL algorithm for non-Markovian, non-stationary reward streams.
problem Maximizing rewards from non-Markovian, non-stationary reward streams.
method Uses causal DAG to construct Markov states, solves periodic MDP.
result Optimal state construction maximizes discounted rewards.
Actor-Critic method achieves optimal regret for unichain MDPs.
problem Scalable regret analysis for infinite-horizon average-reward MDPs.
method NAC-B, a Natural Actor-Critic with batching.
result Order-optimal regret of i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) in infinite-horizon average-reward MDPs. The paper tackles resource allocation for arms with unknown and random rewards, achieving optimal regret bounds.
problem Allocating resources on arms with unknown and random rewards.
method Developed two algorithms with optimal regret bounds for b ∈ [ 0 , 1 ] b \in [0,1] b ∈ [ 0 , 1 ] , demonstrating a phase transition at b = 1 / 2 b=1/2 b = 1/2 . result Achieved optimal gap-dependent and gap-independent regret bounds for b ∈ [ 0 , 1 ] b \in [0,1] b ∈ [ 0 , 1 ] . We consider Markov Decision Processes (MDPs) where the rewards are unknown and may change in an adversarial manner. We provide an algorithm that achieves state-of-the-art regret bound of O ( τ ( ln ∣ S ∣ + ln ∣ A ∣ ) T ln ( T ) ) O( \sqrt{τ(\ln|S|+\ln|A|)T}\ln(T)) O ( τ ( ln ∣ S ∣ + ln ∣ A ∣ ) T ln ( T )) , where S S S is the state space, A A A is the action space, τ τ τ is the mixing time of the MDP, and $…
Optimizes trading strategy considering alpha decay and transaction costs.
problem Maximizing reward in a multi-period portfolio with transaction costs and alpha decay.
method Formulated as an infinite horizon Markov Decision Process, solved using a modified value iteration algorithm with convergence proof and asymptotic analysis.
result Characterized optimal trading policy that maximizes average expected reward.
Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward…
Study optimal portfolio strategies with periodic evaluation under short-selling prohibition.
problem Optimal portfolio strategies with periodic evaluation under short-selling prohibition.
method Reformulate the original problem into an auxiliary one-period optimization problem and introduce dual control problem.
result Derive and verify the value function and optimal constrained portfolio for the original problem.
Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the a…
Study non-linear combinatorial bandits with polynomial rewards, finding significant differences from linear cases.
problem Adversarial combinatorial bandits with general non-linear reward functions.
method Extending existing work on adversarial linear combinatorial bandits, analyzing minimax optimal regret for polynomial and non-polynomial reward functions.
result Minimax optimal regret bounds for adversarial combinatorial bandits with general non-linear reward functions.
New method decomposes Markov chain rewards into persistent and transient components.
problem Ambiguity in classical evaluation methods for Markov chains with reducible and periodic states.
method Minimal exact quotient by the real peripheral invariant subspace, decomposing rewards into persistent and transient components.
result Exact comparison with classical methods shows that the new decomposition reallocates the same information, making persistent modes explicit.
New strategies for identifying the best arm in bandits with decreasing variances.
problem Best arm identification in bandits with time-varying variances.
method Two policies: initial wait followed by continuous sampling, and periodic sampling with weighted average.
result Analytical guarantees and simulations show improved performance over existing methods.
In this paper we present our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning(UL) and Reinforcement Learning(RL) modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for dri…
Study shows how repetition affects learning in bandit settings, providing algorithms with sublinear regret.
problem Effect of persistence of engagement on learning in stochastic multi-armed bandit settings.
method Novel algorithms that achieve sublinear regret under temporal constraints.
result Additive effect of priming on regret upper bound, matching popular algorithms in absence of priming.
New bandit algorithm detects and adapts to seasonal changes in rewards.
problem Adapting to abrupt changes in user preferences during events.
method Detects and adapts to seasonal changes in reward function.
result Outperforms state-of-the-art algorithms for non-stationary environments.
Enhances portfolio performance using deep reinforcement learning and future rewards.
problem Improving existing high-performing portfolio strategies through dynamic rebalancing.
method Proximal Policy Optimization (PPO) and Oracle agents for dynamic rebalancing; Regret-based Sharpe reward function; Transaction cost scheduler; Future-looking reward function; Circular block bootstrap training.
result Significantly enhanced portfolio performance compared to traditional strategies and baselines.
Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
Reinforcement learning for optimizing retirement plans and target dated funds.
problem Optimizing financial goals through periodic investments and withdrawals.
method G-Learner and GIRL algorithms for goal-based wealth management.
result G-Learner provides a computationally tractable solution for wealth management tasks.
Proposes an online model for LLM cascading with adaptive API selection.
problem Adaptive querying and selection of LLM APIs in a context-dependent environment.
method Develops a learning approach combining GMM estimation and UCB-style bounds.
result Achieves cumulative regret of O ~ ( T ) \widetilde O(\sqrt T) O ( T ) over T T T periods. Backtesting framework for CLMMs on Uniswap V3 reduces reward estimation error.
problem Estimating rewards for CLMMs in Uniswap V3 liquidity pools.
method Parametric model for liquidity distribution, historical data analysis.
result Error in reward estimation less than 1% for each pool.
Algorithm solves job acceptance problem with random arrivals and values.
problem Decision-making under random job arrivals and values with limited acceptance.
method Proposes Non-Parametric Sequential Allocation (NPSA) algorithm.
result Expected reward converges to optimality as sample size increases.
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…
Develops neural network framework for risk-reward optimization problems.
problem Multi-period risk-reward optimization with constrained policies.
method Neural network framework with two coupled feedforward networks, parametrizing two-step policies.
result Empirical optimum converges to true optimal value as network capacity and training size increase.
Sequential experiments are often characterized by an exploration-exploitation tradeoff that is captured by the multi-armed bandit (MAB) framework. This framework has been studied and applied, typically when at each time period feedback is received only on the action that was selected at that period. However, in many pr…
Investigates risk measures for DC pension decumulation.
problem Develop optimal decumulation strategies for DC plan holders.
method Formulates decumulation as a control problem, studies risk measures (expected shortfall, linear shortfall, probability of shortfall).
result Optimal controls for expected reward and expected shortfall are identical to those for expected reward and linear shortfall.
This paper uses Thompson sampling to optimize portfolio selection.
problem Difficulty in estimating parameters for Markowitz's mean-variance optimization.
method Portfolio bandit strategy using Thompson sampling.
result Optimal investment portfolio can adapt to different investment periods.
Improved algorithms solve multi-period multi-class packing problems with bandit feedback.
problem Optimizing item packing under budget constraints with class-dependent rewards and bandit feedback.
method Developed a new estimator and a closed-form bandit policy for linear contextual multi-class multi-period packing problems.
result The proposed policy achieves sublinear regret in non-degenerate contexts, significantly outperforming benchmarks.
A contextual bandit method evaluates and improves inventory control policies.
problem Evaluating and improving periodic review inventory control policies with nonstationary demand.
method Contextual bandit-based algorithm to evaluate and tweak policies.
result The method achieves favorable guarantees in both theory and practice.
New RL algorithm tackles nonstationary MDPs with linear approximations and varying rewards.
problem Nonstationary reinforcement learning with evolving reward and state transition functions.
method Developed a new algorithm LSVI-UCB-Restart with periodic restart, and parameter-free Ada-LSVI-UCB-Restart for unknown variation budgets.
result First minimax dynamic regret lower bound for nonstationary linear MDPs and linear MDPs lower bound.
DS-TS adapts to abrupt and smooth changes in bandit problems.
problem Non-stationary multi-armed bandit problems with abrupt and smooth changes.
method Discounted Thompson Sampling with Gaussian priors.
result Achieves nearly optimal regret bound for both abrupt and smooth changes.
How effective are the most common trading models? The answer may help investors realize upsides to using each model, act as a segue for investors into more complex financial analysis and machine learning, and to increase financial literacy amongst students. Creating original versions of popular models, like linear regr…
New method for optimizing risk in financial models using Fourier transforms.
problem Optimizing risk in financial models with multi-period mean-CVaR.
method Strictly monotone 2D integration scheme via Fourier-trained transition kernels.
result Established robust and accurate optimization method for financial models.
PROPO tackles non-stationary MDPs with efficient policy optimization.
problem Non-stationary MDPs with varying reward and transition kernels.
method PROPO, a periodic restarted optimistic policy optimization algorithm with sliding-window-based policy evaluation and improvement.
result PROPO achieves near-optimal performance in non-stationary MDPs.
A new insurance and reinsurance pricing scheme based on realized loss.
problem Determining fair and risk-adjusted insurance premiums.
method Performance-based variable premium scheme with random initial premium adjusted based on realized loss.
result The variable premium scheme reduces reinsurer's total risk exposure compared to expected-value premium.
A federated learning algorithm tackles unknown contexts in multi-arm bandits.
problem Learning optimal actions in federated multi-arm bandits with unobserved contexts.
method Elimination-based algorithm for linearly parametrized reward functions.
result Proved regret bound for linearly parametrized reward functions.
This paper improves Thompson Sampling for complex decision-making problems.
problem Learning in infinite-horizon discounted decision processes with unknown parameters.
method Developed a general canonical probability space and new metrics for analyzing adaptive learning algorithms.
result Thompson Sampling achieves complete learning in complex decision-making problems.
Study uses RL to optimize global equity portfolios, finds mixed results.
problem Optimizing dynamic portfolio weights across diverse global markets.
method Deep reinforcement learning with Soft Actor-Critic, incorporating various constraints and reward formulations.
result RL strategies achieve competitive performance, but no strategy consistently outperforms Buy and Hold.
Adaptive smooth non-stationary bandits achieve optimal regret rates without knowing parameters.
problem Smooth non-stationary bandits with Hölder class rewards.
method Established optimal dynamic regret rate and adaptive algorithm.
result Optimal dynamic regret can be attained adaptively without knowing Hölder exponent and coefficient.
New approach limits regret in non-stationary bandits.
problem Understanding worst case regret in time-varying bandits.
method Belief inertia argument to resist new evidence after changes.
result Linear worst case regret for classical and restarting algorithms.
Paper tackles inventory management with deep learning, improving performance and adherence to constraints.
problem Managing inventory with limited resources and constraints.
method Proposes a novel method to sample from a distribution of possible constraint paths, extends exo-IDP formulation, introduces neural coordinator, and uses modified DirectBackprop algorithm.
result Deep reinforcement learning policies with a neural coordinator outperform classic baselines in terms of performance and adherence to constraints.
New algorithm reduces cold-start costs in multi-armed bandits for many products.
problem High burn-in costs in multi-armed bandits for new products.
method Two-phase bandit algorithm using subsampling and low-rank matrix estimation.
result Reduces burn-in costs and expedites experiment in large product sets.
Paper uses deep reinforcement learning for optimal stock portfolio management.
problem Optimizing stock portfolio choices in complex market environments.
method Direct deep reinforcement learning to learn factor representations and make optimal decisions.
result Deep learning outperforms average market performance in portfolio allocation.
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.