New algorithms ensure fair selection in combinatorial semi-bandit with unrestricted delays.
problem Fair selection in stochastic combinatorial semi-bandit with delayed feedback.
method Introduced merit-based fairness constraints and new bandit algorithms for reward and fairness.
result Achieved sublinear expected reward and fairness regrets with dependence on delay distribution quantiles.
Study uses randomized allocation for delayed rewards in multi-armed bandits.
problem Delayed rewards in contextual multi-armed bandits.
method Randomized allocation with nonparametric estimation.
result Strongly consistent strategy for delayed rewards.
Study explores strategies for randomized allocation in delayed rewards bandits.
problem Understanding the exploration-exploitation tradeoff in randomized strategies with delayed rewards.
method Examines two strategies: updating exploration sequence at every time point vs. updating only when a new reward is observed.
result The strategy updating only when a new reward is observed leads to strong consistency in allocation for a wider scope of situations.
Survey of MAB strategies for non-stationary reward distributions with delayed feedback.
problem Optimizing product availability in an online grocery pick-up platform with non-stationary and delayed reward feedback.
method Evaluation of ε ε ε -greedy, UCB1, Thompson Sampling, and a new adaptive technique (AG1) in MAB simulations. result AG1 outperforms traditional MAB strategies in minimizing regret for non-stationary and delayed feedback.
Adapts two algorithms for online learning with delayed rewards.
problem Online learning with delayed rewards in generalized linear contextual bandits.
method Modifies upper confidence bounds and Thompson sampling algorithms for delayed rewards.
result Both algorithms can be made robust to delays, improving their performance.
We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance p…
A new algorithm tackles delayed combinatorial semi-bandit with causal relations.
problem Optimizing decisions in a non-stationary environment with delayed and causally related rewards.
method Formalized as a non-stationary delayed combinatorial semi-bandit problem, the approach models causal relations with a directed graph in a stationary structural equation model. The agent learns these relations from delayed feedback to optimize decisions.
result Proved a regret bound for the proposed algorithm's performance.
New algorithm reduces regret in delayed feedback generalised linear bandits.
problem Regret in delayed feedback generalised linear bandits.
method Adaptation of optimistic algorithm to delayed feedback.
result Achieves a regret bound independent of the horizon's delay penalty.
We study a variant of the stochastic K K K -armed bandit problem, which we call "bandits with delayed, aggregated anonymous feedback". In this problem, when the player pulls an arm, a reward is generated, however it is not immediately observed. Instead, at the end of each round the player observes only the sum of a number…
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
GASIL encourages agents to imitate past good trajectories in reinforcement learning.
problem Long-term credit assignment in sparse and delayed reward environments.
method Generative Adversarial Imitation Learning (GASIL) framework.
result GASIL improves performance in reinforcement learning tasks with delayed rewards.
New MAB problem with delayed, anonymous feedback analyzed.
problem Delayed, anonymous feedback in stochastic bandits.
method Phase-based extensions of UCB algorithm for SDCAF.
result Sub-linear regret guarantees for proposed algorithms.
New algorithm tackles delayed feedback in Lipschitz bandits with sublinear regret.
problem Delayed feedback in Lipschitz bandits.
method Design of algorithms for bounded and unbounded stochastic delays.
result Sublinear regret guarantees for both bounded and unbounded delays.
New algorithm for multi-armed bandits with delayed, partially observed rewards.
problem Sequential decision-making with delayed feedback.
method Proposed multi-armed bandits with generalized temporally-partitioned rewards, introducing β-spread property.
result Upper bound on performance of TP-UCB-FR-G algorithm improves state of the art.
This paper proposes a new algorithm for learning guidance rewards in RL.
problem Long-term temporal credit assignment in sparse or delayed reward environments.
method Surrogate RL objective with trajectory-space smoothing to learn guidance rewards.
result Guidance rewards can be learned without additional neural networks and have intuitive interpretation.
The paper develops a reinforcement learning model to estimate ad impact considering delayed and cumulative effects.
problem Accurately estimating ad impact considering delayed and long-term effects, cumulative impacts, and customer heterogeneity.
method Modeling ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards, proposing a two-stage maximum likelihood estimator and reinforcement learning algorithm.
result Achieves a near-optimal regret bound of O ~ ( d H 2 T ) \tilde{O}{(dH^2\sqrt{T})} O ~ ( d H 2 T ) , validating the approach through simulation experiments. A novel bandit problem with delayed arms, showing optimal strategies and lower bounds.
problem Optimizing reward in a stochastic multi-armed bandit setting with delayed arms.
method Mapping to PINWHEEL scheduling problem, simple greedy algorithm, UCB based algorithm, lower bounds.
result Simple greedy algorithm is asymptotically ( 1 − 1 / e ) (1-1/e) ( 1 − 1/ e ) optimal and UCB based algorithm has c log T + o ( log T ) c \log T + o(\log T) c log T + o ( log T ) cumulative regret. Align-RUDDER improves reinforcement learning with few demonstrations by redistributing rewards.
problem Learning complex tasks with sparse and delayed rewards using few demonstrations.
method Align-RUDDER uses a profile model for reward redistribution based on multiple sequence alignment of demonstrations.
result Align-RUDDER outperforms competitors on complex tasks with few demonstrations.
Paper tackles action delays in reinforcement learning, proposing a delay-aware framework.
problem Action delays degrade reinforcement learning performance in real-world systems.
method Formal definition of delay-aware MDP, transformation into standard MDP with augmented states, delay-aware model-based reinforcement learning framework.
result Proposed framework is more efficient in training and transferable between systems with various delay durations.
New algorithm optimizes long-term user satisfaction in recommendation systems.
problem Optimizing long-term user satisfaction in recommendation systems with delayed rewards.
method Developed a predictive model of delayed rewards and a bandit algorithm that balances exploration and exploitation.
result Our approach results in substantially better performance compared to short-term or delayed optimization.
Study cooperative bandit learning with imperfect communication, achieving near-optimal performance.
problem Real-world distributed decision-making with imperfect communication.
method Proposed decentralized algorithms for three communication scenarios: stochastic networks, random delays, and adversarially corrupted rewards.
result Achieved competitive performance and near-optimal guarantees on group regret.
Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (M…
Algorithm improves RL by discovering delayed causal relations.
problem Improving data-efficiency and interpretability in RL.
method Predicts observations with Markov assumption, introduces hidden variables to explain past events.
result Significantly improves RL performance on simulated and real tasks.
RAM extends attention-based mechanism for existence determination.
problem Binary determination of object existence in cluttered images.
method Recurrent attention model (RAM) with k k k -maximum aggregation layer and new reward mechanism. result Significant efficiency and accuracy improvement over existing approaches.
A new algorithm for cryo-EM data collection that balances reward and latency.
problem Optimizing data collection in cryo-EM experiments with action delays.
method Latency-aware contextual bandit framework and COAF algorithm.
result The COAF algorithm efficiently maximizes reward over time in cryo-EM experiments.
A Q-learning approach optimizes RTB ad campaigns for mobile app installs.
problem Optimizing RTB ad campaigns for mobile app installs with delayed rewards.
method State space based policy trained via Q-learning algorithm to handle delayed install notifications.
result Significant increase in profit and number of efficient campaigns.
New algorithm for bandits with delayed action effects, reducing regret.
problem Delayed impact of actions in multi-armed bandits.
method Formulated a new bandit setting with delayed action effects, proposed an algorithm with regret bound.
result Achieved a regret of i l d e O ( K T 2 / 3 ) ilde{\mathcal{O}}(KT^{2/3}) i l d e O ( K T 2/3 ) and showed a matching lower bound. A decentralized algorithm minimizes regret in a network of agents playing stochastic bandits.
problem Minimizing regret in a network of agents playing stochastic bandits with delayed information.
method Fully decentralized algorithm using accelerated consensus and UCB for delayed estimates.
result Regret bound is the optimal centralized regret plus a term depending on spectral gap of communication matrix.
Thompson Sampling tackles noisy context in stochastic bandits.
problem Designing an action policy for noisy, corrupted contexts in stochastic bandits.
method Introducing a Thompson Sampling algorithm for Gaussian bandits with Gaussian context noise, adopting an information-theoretic analysis.
result Demonstrates the Bayesian regret of the proposed algorithm concerning the oracle's action policy.
EBRM improves robustness and generalization of language model rewards.
problem Challenges in capturing complex human preferences and generalizing to unseen data in reward models.
method Energy-Based Reward Model (EBRM) that models reward distribution explicitly, using conflict-aware data filtering, label-noise-aware contrastive training, and hybrid initialization.
result Significant improvements in robustness and generalization, up to 5.97% improvement in safety-critical alignment tasks.
New model for music streaming recommends songs based on past play history.
problem Nonstationary stochastic bandit model with delay-dependent rewards.
method Ranking policies approximating optimal policy with bounded regret.
result Algorithm with O ~ ( k T ) \widetilde{\mathcal{O}}\big(\!\sqrt{kT}\big) O ( k T ) regret and O ( k ln ln T ) \mathcal{O}\big(k\ln\ln T\big) O ( k ln ln T ) switches. Game-theoretic analysis of mining gaps in blockchain systems.
problem Strategic mining behavior and its impact on blockchain stability.
method Game-theoretic model and Nash equilibrium analysis.
result Mining gaps can destabilize blockchain systems, especially with decreasing block rewards.
Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functio…
Algorithm learns optimal coordination for strategic agents in uncertain settings.
problem Optimizing rewards for strategic agents with private types and actions.
method Combines delaying mechanism, reward angle estimation, and LinUCB algorithm.
result Near optimal regret bound of O ~ ( T ) \tilde{O}(\sqrt{T}) O ~ ( T ) for learning optimal policy. The paper introduces return parity for fairness in MDPs, addressing delayed and adverse effects.
problem Fairness in MDPs for dynamic domains with delayed and adverse effects.
method Proposes return parity, decomposes return disparity, and develops algorithms for state visitation distributional alignment.
result The proposed algorithms can successfully close the disparity gap while maintaining policy performance.
The paper proposes a method to decompose value functions in RL for better understanding and prediction.
problem Understanding and predicting the dynamics and returns in reinforcement learning models.
method A two-step approach decomposing the value function into future dynamics and trajectory returns, with a practical deep RL algorithm.
result The proposed algorithm outperforms in MuJoCo tasks, especially under delayed reward settings.
New algorithm tackles multi-agent bandits with heavy-tailed data.
problem Maximizing system performance in multi-agent settings with heavy-tailed data.
method Algorithm exploits hub-like structures and synchronization among clients.
result Regret bound of O ( M 1 − 1 α log T ) O(M^{1 -\frac{1}α} \log{T}) O ( M 1 − α 1 log T ) for homogeneous settings, O ( M log T ) O(M \log{T}) O ( M log T ) for heterogeneous. This paper explores how environmental properties can simplify reinforcement learning in non-episodic settings.
problem Challenges in reinforcement learning with continuous interaction and sparse delayed rewards.
method Analysis of environment shaping and dynamism properties to simplify learning.
result Properties like environment shaping and dynamism can significantly ease learning in non-episodic, sparse reward settings.
Combines Hebbian and DQN for better POMDP problem solving.
problem Difficult POMDP problems with TD errors.
method Modulated Hebbian plus Q network architecture (MOHQA) integrating Hebbian and DQN.
result Improved DQN performance and outperformed other algorithms on some POMDPs.
This paper tackles hard exploration in the game Pommerman, improving RL learning.
problem Hard exploration in sparse, delayed, and deceptive reward domains.
method Developed a model-based automatic reasoning module to prune unsafe actions.
result Model-based approach significantly improves RL learning in Pommerman.
Paper tackles constrained bandit problems with a new learning framework.
problem Optimizing a black-box reward function subject to a black-box constraint function over a continuous space.
method Rectified Pessimistic-Optimistic Learning (RPOL) framework, incorporating optimistic and pessimistic GP bandit learning.
result RPOL achieves sublinear regret and minimal cumulative constraint violation.
Develops a stochastic approach to financial market delays.
problem Modeling delays in financial markets with multiple assets.
method Introduces a general stochastic framework for information and order execution delays.
result Delayed markets maintain fundamental asset pricing theorems and no asymptotic free lunch condition.
New algorithm optimizes multi-objective outcomes in uncertain environments.
problem Optimizing global concave rewards in online Markov decision processes with multiple actions.
method No-regret algorithm based on online convex optimization and UCRL2, with a gradient threshold procedure.
result Non-stationary policy diversifies outcomes to optimize the global concave reward.
A new method eliminates reward estimation variance in sequential decision processes.
problem High variance in gradient estimation hinders sample efficiency in reinforcement learning.
method Proposes an unbiased method that completely eliminates variance under certain conditions.
result The proposed method significantly improves performance in challenging problems with delayed rewards.
Deep learning enables a robot to navigate autonomously within a perimeter.
problem Autonomous path planning for space exploration robots.
method Deep reinforcement learning with randomized reward function parameters.
result Trained robot can navigate to any point within a perimeter without prior knowledge.
Banker-OMD improves online learning with delayed feedback.
problem Handling delayed feedback in online learning.
method Generalized Online Mirror Descent (OMD) framework.
result Achieves nearly-optimal performance in three bandit scenarios.
We address the problem of learning in an online setting where the learner repeatedly observes features, selects among a set of actions, and receives reward for the action taken. We provide the first efficient algorithm with an optimal regret. Our algorithm uses a cost sensitive classification learner as an oracle and h…