DDO-RM improves reward-based policies by converting reward scores into a target distribution.
problem Improving reward-based policies when the reward function is simpler than the policy.
method Converts reward scores into a target distribution and uses KL-regularized mirror-descent updates.
result DDO-RM outperforms DPO in pair accuracy and mean margin.
Paper generalizes reward distribution in multi-armed bandits with temporally-partitioned rewards.
problem Handling partial rewards distributed over multiple rounds in multi-armed bandits.
method Introduces Beta-spread property to generalize reward distribution, derives lower bound, and provides TP-UCB-FR-G algorithm.
result Improves regret upper bound for some scenarios using Beta-spread property.
BelMan uses Bayesian methods to optimize decisions in multi-armed bandit problems.
problem Optimizing decisions in multi-armed bandit problems with varying rewards and beliefs.
method BelMan uses a geometric approach with information projection and reverse projection to balance exploration and exploitation.
result BelMan outperforms other algorithms in specific scenarios involving many arms and continuous rewards.
Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.
problem Reward collapse in aligning large language models with human preferences.
method Introduced a prompt-aware optimization scheme to derive closed-form expressions for reward distributions.
result Our prompt-aware utility functions significantly alleviate reward collapse during training.
Iterative tilting fine-tunes diffusion models for reward-tilted distributions.
problem Fine-tuning diffusion models for reward-tilted distributions.
method Decomposes large reward tilts into smaller, tractable tilts via first-order Taylor expansion, avoiding backpropagation.
result Validated on a two-dimensional Gaussian mixture, achieving exact closed-form solutions.
Bayesian nonparametric models improve multi-armed bandit performance with uncertain rewards.
problem Reward model uncertainty in multi-armed bandits.
method Bayesian nonparametric Gaussian mixture models for flexible reward density estimation.
result Achieves successful regret performance with asymptotic regret bound.
The paper proves the convergence of Q-value for Gaussian rewards.
problem Existing proofs cannot guarantee convergence of the Q-function for Gaussian rewards.
method Using the central limit theorem and relaxing the condition to E[r(s,a)2]<∞. result Proves the convergence of the Q-function under the condition of E[r(s,a)2]<∞. Learning reward functions can lead to poor policy performance despite low error.
problem Low error in learned reward functions does not guarantee low regret in policy performance.
method Mathematical analysis of reward learning and policy optimization.
result A low expected test error of the reward model guarantees low worst-case regret, but error-regret mismatch can occur with certain data distributions.
New algorithm reduces worst-case regret for heavy-tailed bandits.
problem Stochastic Multi-Armed Bandit problem with heavy-tailed rewards.
method Modified minimax policy MOSS with saturated empirical mean.
result Worst-case regret matching lower bound for heavy-tailed distributions.
New algorithms for multivariate RL improve decision-making in complex systems.
problem Complex multi-objective decision-making in reinforcement learning.
method Oracle-free and computationally-tractable algorithms for multivariate distributional RL.
result Convergence rates match scalar reward settings and provide insights into reward dimensionality.
Paper establishes DRL for high-dimensional rewards.
problem Intractable reinforcement learning with high-dimensional rewards.
method Theoretical foundations and a novel DRL algorithm.
result Bellman operator contraction in high-dimensional spaces.
Fine-tunes diffusion models to generate diverse samples with high genuine rewards.
problem Reward collapse in finetuning diffusion models.
method Entropy-regularized control against pretrained diffusion models.
result Efficient generation of diverse samples with high genuine rewards.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
A new policy for contextual bandits adapts to reward vector shifts.
problem Learning under reward vector shifts with ordered rewards.
method Adaptive-discretization and optimistic elimination policy.
result Established upper bounds on preference-based regret.
Language models perform worse with implicit reward models than explicit ones.
problem Understanding why implicit reward models generalize worse than explicit ones.
method Investigated the root cause of the generalization gap between IM-RMs and EX-RMs.
result Implicit reward models rely more on superficial token-level cues, leading to worse generalization.
This paper tackles distribution shift in model-based offline RL, proposing a shifts-aware reward method.
problem Distribution shift challenges model-based offline RL by distorting value estimation and policy optimization.
method The paper disentangles the problem into model bias and policy shift, proposing a shifts-aware reward through probabilistic inference.
result The proposed shifts-aware reward method effectively mitigates distribution shift and improves policy optimization.
Agents cooperate to make decisions in multi-armed bandits over a graph.
problem Optimizing decisions in multi-agent multi-armed bandits with shared information.
method Designs consensus-based distributed estimation and cooperative algorithms for group decision-making.
result Achieves group performance close to centralized fusion center.
New bandit algorithms improve sparse reward learning.
problem Sparse rewards hinder learning efficiency in real-world bandit applications.
method Developed algorithms based on Upper Confidence Bound and Thompson Sampling for zero-inflated distributions.
result Empirical performance of new algorithms is superior to existing methods.
New Thompson sampling algorithm reduces regret for exponential family bandits.
problem Minimizing regret in multi-armed bandit problems with exponential family rewards.
method Proposes ExpTS and ExpTS+ algorithms using novel sampling distributions. result Minimizes both finite-time and asymptotic regret for exponential family rewards.
EPIC quantifies reward differences without policy optimization.
problem Distinguishing reward function quality from policy optimization issues.
method EPIC distance to compare reward functions directly.
result EPIC bounds policy training success and regret.
Paper designs a bandit algorithm without reward distribution info.
problem Designing bandit algorithms without reward distribution info.
method Alternates between greedy rule and forced exploration.
result Achieves substantial regret upper bounds.
Best-of-N sampling reveals reward targets from preference data, influencing N and base distribution choices.
problem Understanding reward extraction from Best-of-N preference data and optimal N and base distribution choices.
method Specialized analysis of preference data via induced conditional distribution, deriving reward targets and design principles.
result Reward targets are explicit functions of N and base distribution, and bounded-class minimizers approach these targets as N grows.
SDPG algorithm improves sample efficiency and reward in DRL for continuous action spaces.
problem Improving sample efficiency and reward in distributional reinforcement learning for continuous action spaces.
method SDPG algorithm models return distribution using samples via reparameterization technique.
result SDPG shows better sample efficiency and higher reward in OpenAI Gym environments.
Thompson Sampling shows polynomial regret for combinatorial semi-bandits with subgaussian rewards.
problem Finding optimal solutions in combinatorial semi-bandits with suboptimal sampling.
method Proposes Thompson Sampling with polynomial regret for linear combinatorial semi-bandits.
result Demonstrates 'mismatched sampling paradox' where knowing distributions can lead to worse performance.
Motivated by problems in search and detection we present a solution to a Combinatorial Multi-Armed Bandit (CMAB) problem with both heavy-tailed reward distributions and a new class of feedback, filtered semibandit feedback. In a CMAB problem an agent pulls a combination of arms from a set {1,...,k} in each round, g…
A GAN-based approach for multivariate policy evaluation and exploration in reinforcement learning.
problem Multivariate reinforcement learning with high-dimensional data.
method Formulating DiRL as a GAN model for multivariate rewards and state transitions.
result Unified approach for learning values and states with a discrepancy-driven exploration method.
Study noisy rewards in online decision-making with unknown distributions.
problem Learning optimal decisions in online settings with noisy and unknown reward distributions.
method Proposes algorithms integrating learning and decision-making via LCB thresholding.
result Achieves competitive ratios of 1 - 1/e and 1/2 in various settings.
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method Intrinsic rewards proportional to the entropy of future state-action features.
result The new objective leads to improved visitation of features within individual trajectories.
New RL approach uses future state and action visitation measures for better exploration.
problem Improving exploration in reinforcement learning.
method Intrinsic reward based on future state and action visitation measures, using contraction operators.
result Policies achieve good state-action space coverage and high performance.
New algorithms optimize risk-aware selection in uncertain rewards.
problem Balancing expected reward and risk in uncertain, potentially heavy-tailed rewards.
method Distribution oblivious algorithms that consider CVaR, not bound on moments/tails.
result Provable upper bounds on incorrect identification probability.
Training-free method improves large language model sequence quality via reward-guided sampling.
problem Optimizing large language model sequence quality over token likelihood.
method Reward-augmented target distribution combined with Sequential Monte Carlo sampling.
result Significant gains in sequence generation and mathematical reasoning tasks.
New algorithms handle heavy-tailed rewards in reinforcement learning.
problem Learning from heavy-tailed rewards in reinforcement learning.
method Robust mean estimation techniques for constructing algorithms.
result Near-optimal regret bounds achieved in heavy-tailed reward settings.
DSAC improves robustness of imitation learning.
problem Learning from few expert data and distribution shift.
method DSAC uses adversarial reward function.
result DSAC performs well on PyBullet environments.
We implement momentum strategies using reward-risk measures as ranking criteria based on classical tempered stable distribution. Performances and risk characteristics for the alternative portfolios are obtained in various asset classes and markets. The reward-risk momentum strategies with lower volatility levels outper…
The paper explores how mining costs, rewards, and blockchain security are interconnected.
problem Understanding the interdependencies between mining costs, mining rewards, and blockchain security.
method Theoretical derivation and empirical analysis using daily crypto market data and autoregressive distributed lag approach.
result Cryptocurrency price and mining rewards are intrinsically linked to blockchain security outcomes.
Study on identifying the best decision with continuous, separable rewards in stochastic multi-armed bandit.
problem Identifying the best decision with continuous, separable rewards in stochastic multi-armed bandit.
method Proposed an adaptive learning algorithm for CPE-CS problem, analyzed sample complexity with new hardness measure.
result Upper and lower bounds of sample complexity for CPE-CS problem.
New algorithm for shareable arms with load-dependent rewards in stochastic bandits.
problem Learning optimal play strategy with shareable finite-capacity arms in stochastic bandits.
method Developed a capacity estimator and online learning algorithm for MP-MAB with shareable arms.
result Regret upper bound matches the lower bound, validating the algorithm's performance.
Survey of MAB strategies for non-stationary reward distributions with delayed feedback.
problem Optimizing product availability in an online grocery pick-up platform with non-stationary and delayed reward feedback.
method Evaluation of ε-greedy, UCB1, Thompson Sampling, and a new adaptive technique (AG1) in MAB simulations. result AG1 outperforms traditional MAB strategies in minimizing regret for non-stationary and delayed feedback.
New adaptive methods improve dynamic Bernoulli bandit performance.
problem Optimizing rewards in dynamic environments with changing reward distributions.
method Adaptive estimation in standard bandit algorithms (ε-Greedy, UCB, Thompson sampling).
result New algorithms outperform standard methods in dynamic environments.
Unified approach to time-inconsistent problems with distribution-dependent rewards.
problem Time-inconsistent problems with distribution-dependent rewards in behavioral finance and economics.
method Equilibrium master equation on Wasserstein space, refined derivatives, Itô's formula.
result Unified approach to find equilibrium solutions for time-inconsistent problems.
New method uses SMC for Bayesian MABs with time-varying, nonlinear rewards.
problem Learning optimal actions in dynamic, non-stationary environments.
method Sequential Monte Carlo (SMC) for estimating statistics of unknown reward distributions.
result SMC-based Bayesian MAB policies achieve good regret performance in non-stationary, nonlinear reward scenarios.
SIRL recovers reward function probability distribution from expert actions.
problem Recovering reward functions from expert demonstrations in reinforcement learning.
method Monte Carlo Expectation-Maximization (MCEM) method to estimate reward function probability distribution.
result SIRL provides a robust and transferable solution to the IRL problem.
Model shows how centralization occurs in cryptocurrency mining.
problem Centralization of reward and computational power in Bitcoin-like cryptocurrencies.
method Mean field game model to study miner competition and reward distribution.
result Heterogeneity of initial wealth leads to greater imbalance in reward distribution.
This work analyzes the value of future reward information in RL.
problem Analyzing the impact of knowing future rewards in reinforcement learning.
method Competitive analysis and worst-case reward distribution.
result Exact ratios between standard RL agents and those with future-reward lookahead.
Variational inference improves Thompson sampling for complex contextual bandit problems.
problem Optimizing decisions in scenarios with unknown and complex reward distributions.
method Applying variational inference to approximate complex reward distributions in Thompson sampling.
result The proposed variational Thompson sampling achieves reduced regrets compared to traditional Thompson sampling.
Improved selection of best outputs from LLMs for better accuracy.
problem BoN fails to reliably find the correct answer with imperfect rewards.
method Majority-of-the-Bests (MoB) selects the mode of a bootstrapped output distribution of BoN.
result MoB consistently improves over BoN in 25 out of 30 setups.
A novel algorithm minimizes regret in a multi-agent bandit problem with time-varying random graphs and heterogeneous rewards.
problem Minimizing regret in a multi-agent multi-armed bandit problem with time-varying random graphs and heterogeneous rewards.
method Introduces a novel algorithmic framework combining averaging-based consensus with a weighting technique and upper confidence bound.
result Derives optimal instance-dependent regret upper bounds of order logT in both sub-gaussian and sub-exponential environments. BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.