This work analyzes the value of future reward information in RL.
problem Analyzing the impact of knowing future rewards in reinforcement learning.
method Competitive analysis and worst-case reward distribution.
result Exact ratios between standard RL agents and those with future-reward lookahead.
New IRL algorithm identifies optimal reward and policy from expert demonstrations.
problem Understanding reward functions from expert demonstrations with neural networks.
method Two-timescale single-loop IRL algorithm for neural network parameterized rewards.
result First IRL algorithm with non-asymptotic convergence guarantee and global optimality in neural network settings.
COCOA improves credit assignment in reinforcement learning by measuring contributions to rewards.
problem Improving sample efficiency in reinforcement learning through better credit assignment methods.
method Counterfactual Contribution Analysis (COCOA) for precise credit assignment.
result COCOA achieves lower bias and variance compared to Hindsight Credit Assignment (HCA), improving reinforcement learning performance.
This paper analyzes how randomizing rewards in MBRL can improve performance without being overly optimistic.
problem The gap between theoretical worst-case regret analysis and empirical performance in MBRL.
method Reward randomization in model-based reinforcement learning (MBRL) with kernelized linear regulator (KNR) model.
result Reward randomization guarantees partial optimism and near-optimal worst-case regret.
Paper analyzes IRL problem and provides sample complexity analysis.
problem Finding a reward function for a given optimal policy.
method Geometric analysis and L1-regularized SVM formulation.
result Sample complexity of O(n^2 log(nk)) for recovering reward function.
New insights into balancing reward and fairness in stochastic MAB.
problem Balancing reward and fairness in stochastic multi-armed bandits.
method Formulated a penalization framework and proposed a hard-threshold UCB-like algorithm.
result Asymptotic fairness, nearly optimal regret, better reward-fairness tradeoff.
This paper introduces a gradient analysis framework to improve language model performance by rewarding good examples and penalizing bad ones.
problem Improving language model output quality by penalizing bad examples.
method Gradient analysis of loss functions to reward good examples and penalize bad ones.
result ExMATE is superior to MLE and combining DPO with ExMATE enhances performance.
SIBRE boosts reinforcement learning convergence by rewarding improvement over past performance.
problem Improving the rate of convergence in reinforcement learning.
method SIBRE is a reward shaping approach that rewards improvement over the agent's own past performance.
result SIBRE converges faster and more stably to the optimal policy compared to baseline RL algorithms.
This work analyzes nonexpansive stochastic approximations with Markovian noise, proving convergence in reinforcement learning.
problem Applying stochastic approximation to reinforcement learning settings with nonexpansive operators.
method Investigates nonexpansive stochastic approximations with Markovian noise, providing asymptotic and finite sample analysis.
result First-time proof of convergence for classical tabular average reward temporal difference learning.
Incentivized exploration improves MAB performance under reward drift.
problem Improving exploration in multi-armed bandits with biased feedback.
method Analysis of three MAB algorithms (UCB, ε-Greedy, Thompson Sampling) under drifted reward feedback.
result All algorithms achieve O ( log T ) \mathcal{O}(\log T) O ( log T ) regret and compensation under drifted reward. Improved inference-time alignment using Best-of-N and smoothing.
problem Reward overoptimization in Best-of-N (BoN) due to poor proxy reward models.
method Introduced Soft Best-of-N (SBoN) and analyzed its performance through KL divergence and regret analysis.
result Smoothing helps SBoN mitigate reward overoptimization, especially when proxy reward quality is low.
A fair reward system boosts participation in federated learning.
problem Fairness in federated learning among competitive agents with siloed data.
method Hierarchically fair federated learning (HFFL) framework with proportional rewards based on contribution levels.
result Efficacy of HFFL in maintaining fairness and facilitating federated learning in competitive settings.
PHE adds pseudo-rewards to history to minimize regret in stochastic bandits.
problem Minimizing cumulative regret in stochastic multi-armed bandits.
method PHE algorithm that adds O ( t ) O(t) O ( t ) i.i.d. pseudo-rewards to history and pulls the best arm based on the perturbed history. result Near-optimal regret bounds derived for PHE.
Algorithm improves multi-armed bandit performance by transferring reward samples.
problem Sequential multi-armed bandit problem with changing reward distributions.
method UCB algorithm with reward sample transfer.
result Significant improvement in cumulative regret over standard UCB.
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
Diversified risk parity strategies outperform equally-weighted portfolios in various asset universes.
problem Finding optimal portfolio allocations that balance risk and reward.
method Integrates various reward-risk measures and generic allocation rules into diversified risk parity.
result Diversified reward-risk parity strategies exhibit higher average returns, Sharpe ratios, and Calmar ratios compared to equally-weighted risk portfolios.
RUDDER simplifies Q-value estimation for delayed rewards in MDPs.
problem Solving delayed rewards in reinforcement learning with bias and variance issues.
method Reward redistribution and return decomposition to simplify Q-value estimation.
result RUDDER significantly speeds up Q-value estimation and improves performance on Atari games.
Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.
problem Non-uniqueness of reward functions in inverse reinforcement learning.
method Entropy regularization combined with least-squares reconstruction of the reward from the soft Bellman residual.
result Least-squares reward function is unique and consistent with the expert policy.
The paper analyzes errors in terminating reinforcement learning prematurely.
problem Premature termination of reinforcement learning leads to significant errors.
method Combinatorial analysis and simulation experiments.
result The error of premature termination can be quite high and cannot be eliminated.
Actor-Critic method achieves optimal regret for unichain MDPs.
problem Scalable regret analysis for infinite-horizon average-reward MDPs.
method NAC-B, a Natural Actor-Critic with batching.
result Order-optimal regret of i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) in infinite-horizon average-reward MDPs. Bayesian REX learns Atari games from demonstrations efficiently.
problem Bayesian reward learning for complex control problems is computationally intractable.
method Bayesian Reward Extrapolation (Bayesian REX) pre-trains a low-dimensional feature encoding and uses preferences to perform fast Bayesian inference.
result Bayesian REX learns Atari games from demonstrations in 5 minutes, competitive with state-of-the-art methods.
New MAB problem with delayed, anonymous feedback analyzed.
problem Delayed, anonymous feedback in stochastic bandits.
method Phase-based extensions of UCB algorithm for SDCAF.
result Sub-linear regret guarantees for proposed algorithms.
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
Study optimal adaptive allocation for multi-armed bandits with Markovian rewards.
problem Optimal adaptive allocation for multi-armed bandits with Markovian rewards.
method Round-robin Kullback-Leibler upper confidence bounds for optimal adaptive allocation.
result Logarithmic dependence of regret on time horizon, asymptotically optimal.
B-REX efficiently learns Atari game policies from pixel inputs using Bayesian methods.
problem Learning reward functions from visual inputs with uncertainty and safety considerations.
method Bayesian Reward Extrapolation (B-REX) using successor features and preferences.
result B-REX generates posterior samples efficiently, enabling high-confidence performance bounds.
A new framework assesses financial and ESG risks for sustainable investing.
problem Measuring risk and reward in sustainable investing considering environmental, social, and governance factors.
method Proposes axiomatic definitions for ESG-coherent risk measures and reward-risk ratios based on bivariate random variables.
result Empirical analysis ranks stocks using the proposed measures.
Bayesian nonparametric models improve multi-armed bandit performance with uncertain rewards.
problem Reward model uncertainty in multi-armed bandits.
method Bayesian nonparametric Gaussian mixture models for flexible reward density estimation.
result Achieves successful regret performance with asymptotic regret bound.
New algorithms improve efficiency in learning from personalized rewards.
problem Learning from personalized rewards in recommendation systems.
method Developed provably efficient algorithms with sublinear regret for context-dependent feedback.
result Introduced a Lipschitz reward estimator that improves generalization performance.
Paper tackles robustness in reward learning with partial identifiability.
problem Partial identifiability in reward learning leads to unreliable target reward recovery.
method Introduces a robust approach to maximize performance with respect to the worst-case reward in the feasible set.
result Develops Rob-ReL, an algorithm that maximizes performance under worst-case identifiability conditions.
Method explains MDP optimal policies and reward collection.
problem Understanding optimal policies and reward collection in MDPs.
method Deterministic MDPs with a method to map state space and explain rewards.
result Algorithm determines which rewards will be collected and their frequency.
New Thompson sampling algorithm reduces regret for exponential family bandits.
problem Minimizing regret in multi-armed bandit problems with exponential family rewards.
method Proposes ExpTS and ExpTS + ^+ + algorithms using novel sampling distributions. result Minimizes both finite-time and asymptotic regret for exponential family rewards.
We tackle the problem of online reward maximisation over a large finite set of actions described by their contexts. We focus on the case when the number of actions is too big to sample all of them even once. However we assume that we have access to the similarities between actions' contexts and that the expected reward…
Study non-linear combinatorial bandits with polynomial rewards, finding significant differences from linear cases.
problem Adversarial combinatorial bandits with general non-linear reward functions.
method Extending existing work on adversarial linear combinatorial bandits, analyzing minimax optimal regret for polynomial and non-polynomial reward functions.
result Minimax optimal regret bounds for adversarial combinatorial bandits with general non-linear reward functions.
The paper tackles non-stationary MAB with periodic rewards.
problem Non-stationary mean rewards over time in a business context.
method Combines Fourier analysis with confidence-bound learning to estimate periods and minimize regret.
result Proposes a near-optimal policy with a regret bound of O ( T ∑ k = 1 K T k ) O(\sqrt{T\sum_{k=1}^K T_k}) O ( T ∑ k = 1 K T k ) . Paper develops robust estimators and strategies for stochastic MABs with heavy-tailed rewards.
problem Stochastic multi-armed bandits with heavy-tailed rewards.
method Proposes a novel robust estimator and perturbation-based exploration strategy.
result Develops upper and lower regret bounds for various perturbations.
Survey of MAB strategies for non-stationary reward distributions with delayed feedback.
problem Optimizing product availability in an online grocery pick-up platform with non-stationary and delayed reward feedback.
method Evaluation of ε ε ε -greedy, UCB1, Thompson Sampling, and a new adaptive technique (AG1) in MAB simulations. result AG1 outperforms traditional MAB strategies in minimizing regret for non-stationary and delayed feedback.
New algorithms for multivariate RL improve decision-making in complex systems.
problem Complex multi-objective decision-making in reinforcement learning.
method Oracle-free and computationally-tractable algorithms for multivariate distributional RL.
result Convergence rates match scalar reward settings and provide insights into reward dimensionality.
Game-theoretic analysis of mining gaps in blockchain systems.
problem Strategic mining behavior and its impact on blockchain stability.
method Game-theoretic model and Nash equilibrium analysis.
result Mining gaps can destabilize blockchain systems, especially with decreasing block rewards.
Reinforcement learning mimics expert behavior.
problem Learning from expert demonstrations in reinforcement learning.
method Reduction to reinforcement learning with a stationary reward.
result Expert reward can be recovered and imitation learning is bounded.
New algorithm uses Whittle index to improve Q-learning for restless bandits.
problem Optimizing decision-making in multiarmed restless bandits with average reward.
method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.
result Demonstrates significant computational gains and excellent empirical performance.
Thompson Sampling bounds for contextual bandits with sub-Gaussian rewards.
problem Improving the performance of Thompson Sampling in contextual bandits with sub-Gaussian rewards.
method Proved comprehensive bounds on Thompson Sampling expected cumulative regret based on mutual information and lifted information ratio for sub-Gaussian rewards.
result Explicit regret bounds for various contextual bandit scenarios.
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.
New algorithms detect changes in non-stationary MABs for better performance.
problem Non-stationary MAB environments where arm reward distributions change over time.
method Modular Detection Augmented Bandit (DAB) procedures with improved performance lower bounds.
result Modular DAB procedures achieve order-optimal regret bounds for various change detectors and bandit algorithms.
Reinforcement learning with trajectory feedback instead of state-action rewards.
problem Frequent feedback not available in practice.
method Extended reinforcement learning algorithms using trajectory feedback for known and unknown transition models.
result Hybrid optimistic-Thompson Sampling algorithm for unknown transition models.
BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.
BERT embeddings improve sequence quality metrics.
problem Measuring the quality of generated sequences against references.
method Employ contextual BERT embeddings for sequence-level reward.
result Contextual embeddings provide a more effective learning signal.
New algorithms improve multi-task bandit performance by transferring reward samples.
problem Sequential multi-task bandit problems with adjacent similar tasks.
method Two UCB-based algorithms that transfer reward samples between tasks.
result Transfer of reward samples reduces overall regret compared to no transfer.