The paper studies reward concentration in MDPs, covering asymptotic and non-asymptotic settings.
problem Reward concentration in Markov Decision Processes (MDPs).
method Unified approach to reward concentration in MDPs, including asymptotic and non-asymptotic bounds.
result Rate-equivalent definitions of regret for learning policies.
Optimizes liquidity provision intervals for profitable AMM participation.
problem Financial losses from poor liquidity provision intervals and reallocation costs.
method Developed a tractable stochastic optimization problem.
result Computes optimal liquidity provision intervals for profitable liquidity concentration.
Paper tackles offline preference-based RL with human feedback.
problem Offline Preference-based Reinforcement Learning with preference feedback.
method Two-step approach: MLE for reward estimation and distributionally robust planning.
result First guarantee for learning any target policy with polynomial samples.
Backtesting framework for CLMMs on Uniswap V3 reduces reward estimation error.
problem Estimating rewards for CLMMs in Uniswap V3 liquidity pools.
method Parametric model for liquidity distribution, historical data analysis.
result Error in reward estimation less than 1% for each pool.
Study on identifying most preferred policy in bandits with vector-valued rewards.
problem Identifying the most preferred policy in bandits with vector-valued rewards.
method Derive a novel lower bound on sample complexity, design the Preference-based Track and Stop (PreTS) algorithm, and derive a new concentration inequality.
result The sample complexity of PreTS is asymptotically tight.
Paper proposes a hybrid RL algorithm that combines offline and online data without needing reward info.
problem How to efficiently use online data to improve RL policies using only offline data.
method A three-stage hybrid RL algorithm that uses reward-agnostic exploration and model-based offline RL.
result The hybrid RL algorithm outperforms both pure offline and pure online RL in sample complexity.
We propose a new online algorithm for cumulative regret minimization in a stochastic linear bandit. The algorithm pulls the arm with the highest estimated reward in a linear model trained on its perturbed history. Therefore, we call it perturbed-history exploration in a linear bandit (LinPHE). The perturbed history is …
We propose a generic, Bayesian, information geometric approach to the exploration--exploitation trade-off in multi-armed bandit problems. Our approach, BelMan, uniformly supports pure exploration, exploration--exploitation, and two-phase bandit problems. The knowledge on bandit arms and their reward distributions is su…
Trajectory-level supervision allows efficient offline reinforcement learning.
problem Offline reinforcement learning
method Developing a statistical theory for offline policy optimization from trajectory-level labels
result Proving a high-probability guarantee of order O ~ ( H 2 C s a ( π ⋆ ) / n ) \widetilde O(H^2\sqrt{C_{sa}(\pi^\star)/n}) O ( H 2 C s a ( π ⋆ ) / n ) Study optimal adaptive allocation for multi-armed bandits with Markovian rewards.
problem Optimal adaptive allocation for multi-armed bandits with Markovian rewards.
method Round-robin Kullback-Leibler upper confidence bounds for optimal adaptive allocation.
result Logarithmic dependence of regret on time horizon, asymptotically optimal.
Paper tackles unknown variances in best-arm identification.
problem Identifying the best arm with unknown variances in Gaussian distributions.
method Two approaches: empirical variance plugging or adapting transportation costs.
result The impact of unknown variances is small on sample complexity.
This paper studies semiparametric contextual bandits, a generalization of the linear stochastic bandit problem where the reward for an action is modeled as a linear function of known action features confounded by an non-linear action-independent term. We design new algorithms that achieve O ~ ( d T ) \tilde{O}(d\sqrt{T}) O ~ ( d T ) regret …
New algorithm tackles heavy-tailed rewards in RL with instance-dependent regret bounds.
problem Efficient algorithms for RL with heavy-tailed rewards in large state-action spaces.
method Design of \textsc{Heavy-OFUL} for heavy-tailed linear bandits and \textsc{Heavy-LSVI-UCB} for RL with linear function approximation.
result First instance-dependent regret bounds for heavy-tailed rewards in RL with linear function approximation.
New axioms justify ES without NRC, linking it to mean-ES portfolio selection.
problem Economic axioms for portfolio risk assessment and mean-ES portfolio selection.
method Introducing concentration aversion as an alternative to NRC, establishing axiomatic foundations.
result Concentration aversion uniquely characterizes the family of ES and provides new formulas.
OP-GFNs sample candidates in order-preserving proportion to a learned reward function.
problem Sampling diverse candidates with varying rewards in multi-objective optimization.
method Order-Preserving GFlowNets (OP-GFNs) use a learned reward function consistent with a provided order on candidates.
result Training OP-GFNs sparsifies the reward landscape, focusing on higher-ranked candidates.
BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.
Model shows how centralization occurs in cryptocurrency mining.
problem Centralization of reward and computational power in Bitcoin-like cryptocurrencies.
method Mean field game model to study miner competition and reward distribution.
result Heterogeneity of initial wealth leads to greater imbalance in reward distribution.
A new model-free algorithm achieves near-optimal regret for infinite-horizon MDPs.
problem Model-free reinforcement learning for infinite-horizon average-reward MDPs.
method Exploration Enhanced Q-learning (EE-QL) for weakly communicating MDPs.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret bound for general weakly communicating MDPs. The paper examines stability of shares in Proof of Stake protocol, identifying different investor behaviors and phase transitions.
problem Stability of shares in Proof of Stake protocol.
method Identification of large, medium, and small investors under various rewarding schemes; dynamical population model analysis.
result Phase transitions and thresholds for stability are characterized; chaotic centralization leads to concentration of shares.
UCRL3 improves UCRL2's efficiency in reinforcement learning by reducing exploration.
problem Long burn-in phases in numerical experiments of UCRL2.
method UCRL3 uses state-of-the-art time-uniform concentration inequalities and adaptive support computation to tighten exploration.
result UCRL3 achieves a better numerical improvement over UCRL2 in standard environments.
Unified framework for risk-aware policy learning in contextual bandits.
problem Optimizing decision rules in high-stakes domains with adverse outcomes.
method Distributional framework for Lipschitz-continuous risk functionals, with novel empirical concentration inequalities.
result Data-dependent suboptimality bounds with an i l d e O ( 1 / n ) ilde{\mathcal{O}}(1/\sqrt{n}) i l d e O ( 1/ n ) rate, matching risk-neutral offline policy optimization. Demon aligns diffusion models without retraining or backpropagation.
problem Aligning diffusion models with user preferences.
method Stochastic optimization to control noise distribution.
result Significantly improves aesthetics scores for text-to-image generation.
New algorithms reduce regret in neural logistic bandits.
problem Learning unknown reward functions in neural networks.
method Introduced a Bernstein-type inequality for self-normalized vector-valued martingales.
result Regret bounds improved to O ~ ( d ~ κ T ) \widetilde{O}(\widetilde{d}\sqrt{κT}) O ( d κ T ) and O ~ ( d ~ T / κ ) \widetilde{O}(\widetilde{d}\sqrt{T/κ}) O ( d T / κ ) . Motivated by applications of bandit algorithms in education, we consider a stochastic multi-armed bandit problem with ε \varepsilon ε -contaminated rewards. We allow an adversary to give arbitrary unbounded contaminated rewards with full knowledge of the past and future. We impose the constraint that for each time t t t the…
Higher conservative training increases reward-hacking in reasoning models.
problem Reward hacking during online adaptation in reasoning models.
method Conservative offline training with varying levels of conservatism (β) was applied to a Qwen3-14B policy, and online adaptation was measured against a reward ensemble.
result Higher conservatism (β) increases reward-hacking damage, measured by the Goodhart gap and AUGC.
Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several books and surveys. This book provides a more introductory, textbook-like treatment of the subject. Each chapter tackles a p…
The paper tackles resource allocation for arms with unknown and random rewards, achieving optimal regret bounds.
problem Allocating resources on arms with unknown and random rewards.
method Developed two algorithms with optimal regret bounds for b ∈ [ 0 , 1 ] b \in [0,1] b ∈ [ 0 , 1 ] , demonstrating a phase transition at b = 1 / 2 b=1/2 b = 1/2 . result Achieved optimal gap-dependent and gap-independent regret bounds for b ∈ [ 0 , 1 ] b \in [0,1] b ∈ [ 0 , 1 ] . Study of repeated games with unobserved agent rewards using MAB framework.
problem Designing policies for principals in repeated principal-agent games with unobservable agent rewards.
method Developed a policy achieving low regret (square-root regret up to a log factor) for perfect-knowledge agents.
result Constructed an estimator for agent's expected reward and designed a policy achieving low regret.
Self-distillation improves constrained language generation by aligning models with target distributions.
problem Sparse and uninformative reward signals in constrained generation settings.
method Iteratively refining the base model through self-distillation, incorporating learned twist functions and proposals.
result Substantial gains in generation quality through improved model alignment with target distributions.
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
Study non-asymptotic bounds on correlation in high-dimensional linear systems, revealing invariant subspaces and bottlenecks.
problem Understanding correlation and mixing in high-dimensional linear systems with Gaussian noise.
method Sampling from sub-trajectories, using Talagrand's inequality, and analyzing invariant subspaces.
result Large discrepancy between algebraic and geometric multiplicity leads to bottlenecks between invariant subspaces.
We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations…
PDCA algorithm learns policies for RL with constraints using a primal-dual approach.
problem Offline constrained reinforcement learning with general function approximation.
method Primal-Dual-Critic Algorithm (PDCA) using a primal-dual approach.
result PDCA finds a near saddle point of the Lagrangian, nearly optimal for constrained RL.
New algorithm for countable bandits with optimal regret.
problem Stochastic bandit problem with countably many arms.
method Fully adaptive online learning algorithm with O(log n) expected cumulative regret.
result Achieves optimal regret of O(log n) after any number of plays n.
Develops a new model to predict training dynamics of large language models.
problem Lack of mechanistic understanding of training dynamics in large language models.
method A first-principles reduced-order model of training dynamics, predicting group-size invariance and stability thresholds.
result Closed-form model predicts training dynamics with high accuracy and provides new diagnostics.
Classical multi-armed bandit problems use the expected value of an arm as a metric to evaluate its goodness. However, the expected value is a risk-neutral metric. In many applications like finance, one is interested in balancing the expected return of an arm (or portfolio) with the risk associated with that return. In …
Paper introduces risk assessment for contextual bandits without experiments.
problem Evaluate policies using logged data in context bandits.
method Lipschitz risk functionals and Off-Policy Risk Assessment (OPRA) framework.
result OPRA provides finite sample guarantees for various risk estimates.
Survey on risk-aware multi-armed bandits for better decision-making.
problem Risk measures in multi-armed bandits for better decision-making.
method Review of existing research, definition of risk-aware bandit problems, and algorithms for minimizing regret and identifying best arms.
result Consolidation and summarization of existing research on risk measures in multi-armed bandits.
This note examines financial distributions to competing teams at the end of the most famous multiple stage professional (male) bicyclist race, TOUR DE FRANCE. A rank-size law (RSL) is calculated for the team financial gains. The RSL is found to be hyperbolic with a surprisingly simple decay exponent (about equal to -1)…
Develops new methods for risk-aware decision-making in medical bandits.
problem Risk-averse decision-making in medical contexts with limited data.
method Safe, anytime-valid concentration bounds, risk-aware contextual bandits, nonparametric algorithms.
result Improved decision-making algorithms for postoperative patient follow-up.
Optimizes experiment design for causal structure learning in linear models with cycles.
problem Causal structure learning from combined observational and interventional data in linear non-Gaussian cyclic models.
method Combinatorial characterization of equivalence classes, adaptive stochastic optimization, greedy policy with near-optimal performance guarantee, sampling-based estimator for reward function.
result Optimal experiment design reduces the equivalence class of causal graphs to a single true graph with a small number of interventions.
Improved Thompson Sampling using fractional posteriors achieves better regret bounds.
problem Optimizing regret in stochastic multi-armed bandit problems.
method Using α \alpha α -posterior distributions, derived frequentist regret bounds. result Instance-dependent and instance-independent regret bounds established.
New model tackles interference in online experiments.
problem Understanding cumulative performance in interference experiments.
method Introduces Multi-Armed Bandits with Interference (MABI) model.
result Cluster randomization policy achieves optimal expected regret and high probability bound.
Characterizes UCB algorithm performance in bandit problems.
problem Optimizing UCB algorithm performance in stochastic bandit environments.
method Novel perturbation analysis to characterize CLT of pulls and means.
result Smooth interpolation of pseudo-regret between large and small gap regimes.
ACP-UCB1 ranks arms based on upper-tail performance, improving stochastic bandit algorithms.
problem Stochastic bandit algorithms often favor arms with strong upper-tail performance, which is not well-addressed by classical mean-reward criteria.
method ACP-UCB1 combines an adaptive conformal estimate of the upper endpoint with a UCB-type optimism bonus.
result ACP-UCB1 achieves logarithmic upper-quantile regret with per-arm contribution \(O(
icefrac{\log n}{Δ_j^{\mathrm{ACP}}})\).
Theory for algebraic data on categories via concentration structures.
problem Defining algebraic structures on categories.
method Introducing concentration structures and concentration monoids.
result Every group can be represented as a concentration monoid of a trivial category.
This work develops confidence intervals for off-policy evaluation.
problem Estimating expected reward with uncertainty quantification.
method Primal-dual optimization with kernel Bellman loss and martingale concentration inequality.
result Developed practical algorithm for non-asymptotic confidence intervals.
Thompson Sampling improves decision-making in partially observed contexts.
problem Balancing exploration and exploitation in partially observed contextual bandits.
method Thompson Sampling policy for learning optimal arms from noisy linear functions of unobserved context vectors.
result Thompson Sampling achieves poly-logarithmic regret and square-root consistency of parameter estimation.