Study reward-free RL in non-linear settings, improving efficiency and removing assumptions.
problem Improving sample efficiency in reward-free reinforcement learning for non-linear function approximation.
method Proposed RFOLIVE algorithm for minimal structural assumptions, analyzed hardness results for reward-free and reward-aware exploration.
result Statistical efficiency and hardness results under various structural assumptions, no need for reachability or explorability assumptions.
Study non-linear combinatorial bandits with polynomial rewards, finding significant differences from linear cases.
problem Adversarial combinatorial bandits with general non-linear reward functions.
method Extending existing work on adversarial linear combinatorial bandits, analyzing minimax optimal regret for polynomial and non-polynomial reward functions.
result Minimax optimal regret bounds for adversarial combinatorial bandits with general non-linear reward functions.
Extends linear MDP to handle nonlinear rewards.
problem Restrictive linear MDP assumption limits real-world applicability.
method Proposes Generalized Linear MDP (GLMDP) with GLMs for rewards.
result Develops offline RL algorithms achieving suboptimality guarantees.
Reward-free RL in linear MDPs is as hard as reward-aware RL.
problem Reward-free RL in linear MDPs without access to the reward function during exploration.
method Developed a computationally efficient algorithm with sample complexity O ~ ( d 2 H 5 / ε 2 ) \widetilde{\mathcal{O}}(d^2 H^5/ε^2) O ( d 2 H 5 / ε 2 ) . result Achieved optimal d d d dependence in linear MDPs for reward-free RL, matching the reward-aware RL setting. Method predicts future rewards from past actions in a linear Gaussian system.
problem Maximizing cumulative reward in a stochastic multi-armed bandit with linear Gaussian dynamics.
method Proposes a method using a modified Kalman filter to predict future rewards based on past rewards.
result Reward from any action can be used to predict another action's future reward.
New algorithm for reward-free RL with linear function approximation, reducing sample complexity.
problem Efficiently learning optimal policies without prior reward information in complex environments.
method Developed an algorithm for reward-free RL in linear Markov decision processes, proving sample complexity bounds.
result Polynomial sample complexity in feature dimension and planning horizon, independent of states and actions.
DART optimizes subset selection in non-linear bandit problems.
problem Optimizing subset selection in non-linear bandit problems with correlated rewards.
method DART algorithm for combinatorial bandits without individual arm feedback or linearity assumption.
result DART achieves a regret bound of i l d e O ( K K N T ) ilde{\mathcal{O}}(K\sqrt{KNT}) i l d e O ( K K N T ) . New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of O ( d 2 ε − 2 ) O(d^2\varepsilon^{-2}) O ( d 2 ε − 2 ) for finding an ε \varepsilon ε -optimal policy. New algorithms for generalized linear bandits with unknown reward functions.
problem Misspecification of reward functions in existing bandit algorithms.
method Introducing single index bandits, proposing STOR, ESTOR, and GSTOR algorithms.
result Achieved nearly optimal regret bound of i l d e O T ( T ) ilde{O}_T(\sqrt{T}) i l d e O T ( T ) . New algorithm learns optimal policies without explicit rewards.
problem Learning optimal policies without explicit rewards.
method Developed a computationally tractable algorithm for reward-free navigation using linear value iteration.
result PAC guarantees on learning near optimal value functions and policies.
Unified LP framework for offline reward learning from human demonstrations and feedback.
problem Reward learning from human demonstrations and feedback with robustness and sample efficiency.
method A novel linear programming framework for offline reward learning.
result Unified LP framework achieves better performance compared to MLE.
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) regret. The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…
Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquir…
New algorithm learns policies without explicit rewards for MDPs.
problem Learning policies for MDPs without explicit rewards.
method UCRL-RFE algorithm with linear function approximation.
result UCRL-RFE achieves optimal policy with minimal exploration samples.
A new algorithm improves stochastic linear bandit performance using residual bootstrap.
problem Improving performance in stochastic linear bandit problems.
method Residual bootstrap exploration to estimate mean reward and pull the arm with the highest estimate.
result Proposed algorithm exttt{LinReBoot} achieves high-probability sub-linear regret under mild conditions.
Two algorithms tackle heavy-tailed rewards in reinforcement learning with linear function approximation.
problem Online sequential decision-making with heavy-tailed rewards.
method AdaOFUL and VARA algorithms for linear stochastic bandits and MDPs, using modified adaptive Huber regression.
result Achieved state-of-the-art and variance-aware regret bounds for heavy-tailed rewards.
Adaptive policies solve a linear program to maximize rewards while minimizing costs in sales with discounts.
problem Maximizing rewards in sales with discounts while considering costs.
method Solves a linear program based on upper-confidence estimates of conversion probabilities.
result Achieves a regret bound of the typical order (OPT/ B B B ) T \sqrt{T} T , where B is the total budget allowed. New RL algorithm reduces deployment cost for linear function approximations.
problem Efficiently deploying new policies in RL with unknown rewards.
method Proposes an algorithm that minimizes trajectories needed for identifying optimal policies.
result Achieves optimal deployment complexity and sample complexity.
This paper studies semiparametric contextual bandits, a generalization of the linear stochastic bandit problem where the reward for an action is modeled as a linear function of known action features confounded by an non-linear action-independent term. We design new algorithms that achieve O ~ ( d T ) \tilde{O}(d\sqrt{T}) O ~ ( d T ) regret …
Optimizing rewards under budget constraints with correlated costs and rewards.
problem Maximizing total expected reward under a budget constraint on total cost with correlated and potentially heavy-tailed cost-reward pairs.
method Proposes algorithms exploiting correlation between cost and reward via linear minimum mean-square error estimation to achieve tight regret bounds.
result Achieves O ( log B ) O(\log B) O ( log B ) regret for a budget B > 0 B>0 B > 0 under certain moment conditions. This paper studies GAIL's global convergence for general MDP and nonlinear rewards.
problem Understanding when GAIL algorithms achieve global convergence for general MDP and nonlinear rewards.
method Characterization of global convergence for various policy gradient algorithms applied to GAIL.
result First systematic theoretical study of GAIL for global convergence.
IDS optimizes regret in stochastic partial monitoring with linear rewards.
problem Optimizing decision-making in uncertain environments with linear rewards.
method Information Directed Sampling (IDS) for stochastic partial monitoring.
result Achieves optimal regret rates in all observable game regimes.
Research predicts cryptocurrency staking rewards with high accuracy.
problem Predicting cryptocurrency staking rewards.
method Two predictive methodologies: sliding-window average and linear regression models.
result ETH staking rewards can be forecasted with RMSE within 0.7% and 1.1% of the mean value for 1-day and 7-day look-aheads respectively.
UCRL2-VTR achieves nearly optimal regret for learning MDPs with linear function approximation.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method UCRL2-VTR algorithm with Bernstein-type bonus.
result Achieves a regret of i l d e O ( d D T ) ilde{O}(d\sqrt{DT}) i l d e O ( d D T ) with matching lower bound. Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
The paper tackles noisy multi-armed bandit problems with improved regret guarantees.
problem Tackling noisy evaluations in multi-armed bandit problems.
method Derives different algorithmic approaches and theoretical guarantees based on the type of observation functions.
result Improved regret guarantees for noisy linear functions of true rewards.
New IRL algorithm identifies optimal reward and policy from expert demonstrations.
problem Understanding reward functions from expert demonstrations with neural networks.
method Two-timescale single-loop IRL algorithm for neural network parameterized rewards.
result First IRL algorithm with non-asymptotic convergence guarantee and global optimality in neural network settings.
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
Paper improves SLCB regret bound for bounded noise.
problem Stochastic linear contextual bandits with bounded noise.
method Set-membership estimation (SME) and optimism in the face of uncertainty (OFU).
result Improved regret bound of O ( log T ) O(\log T) O ( log T ) . In a linear stochastic bandit model, each arm is a vector in an Euclidean space and the observed return at each time step is an unknown linear function of the chosen arm at that time step. In this paper, we investigate the problem of learning the best arm in a linear stochastic bandit model, where each arm's expected r…
New algorithm reduces regret in delayed feedback generalised linear bandits.
problem Regret in delayed feedback generalised linear bandits.
method Adaptation of optimistic algorithm to delayed feedback.
result Achieves a regret bound independent of the horizon's delay penalty.
A single policy suffices for near-optimal parallel exploration in RL.
problem Quantitative effects of parallel exploration in reward-free RL.
method Using a single policy to guide exploration across all agents.
result Near-linear speedup and near-minimax optimality for linear MDPs.
We study the Combinatorial Pure Exploration problem with Continuous and Separable reward functions (CPE-CS) in the stochastic multi-armed bandit setting. In a CPE-CS instance, we are given several stochastic arms with unknown distributions, as well as a collection of possible decisions. Each decision has a reward accor…
Adapts two algorithms for online learning with delayed rewards.
problem Online learning with delayed rewards in generalized linear contextual bandits.
method Modifies upper confidence bounds and Thompson sampling algorithms for delayed rewards.
result Both algorithms can be made robust to delays, improving their performance.
Iterative tilting fine-tunes diffusion models for reward-tilted distributions.
problem Fine-tuning diffusion models for reward-tilted distributions.
method Decomposes large reward tilts into smaller, tractable tilts via first-order Taylor expansion, avoiding backpropagation.
result Validated on a two-dimensional Gaussian mixture, achieving exact closed-form solutions.
We propose a new online algorithm for cumulative regret minimization in a stochastic linear bandit. The algorithm pulls the arm with the highest estimated reward in a linear model trained on its perturbed history. Therefore, we call it perturbed-history exploration in a linear bandit (LinPHE). The perturbed history is …
INF-clip optimizes heavy-tailed MAB problems with improved performance.
problem Optimizing multi-armed bandit problems with heavy-tailed rewards.
method INF-clip algorithm for adversarial and stochastic heavy-tailed MAB settings.
result INF-clip is optimal for linear and non-linear heavy-tailed stochastic MAB problems.
Contextual multi-armed bandit (MAB) algorithms have been shown promising for maximizing cumulative rewards in sequential decision tasks such as news article recommendation systems, web page ad placement algorithms, and mobile health. However, most of the proposed contextual MAB algorithms assume linear relationships be…
Paper optimizes learning MDPs with adversarial rewards using linear models.
problem Learning MDPs with adversarial rewards and changing reward functions.
method Proposes POWERS, an optimistic policy optimization algorithm.
result Achieves nearly optimal regret bound of O ( d H T ) O(dH\sqrt{T}) O ( d H T ) . Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
LNUCB-TA improves MAB performance by dynamically adjusting exploration rates and recognizing spatiotemporal patterns.
problem Suboptimal performance in environments with rapidly changing reward structures and static exploration rates.
method Hybrid model combining linear and nonlinear estimation, with adaptive k-NN for temporal attention.
result Significantly outperforms state-of-the-art algorithms in cumulative and mean reward, convergence, and robustness.
New algorithms improve linear bandit performance with low computation.
problem Optimizing reward in linear stochastic bandits.
method Reward-biased maximum likelihood method modified for linear and generalized linear bandits.
result New policies achieve order-optimality and competitive empirical performance.
Paper addresses privacy and robustness in stochastic linear bandits.
problem Stochastic linear bandits with differential privacy and adversarial robustness.
method Logarithmic batch queries, arm elimination algorithm, two privacy models.
result First algorithms providing differential privacy and adversarial robustness.
A fair policy for hiring candidates from different groups is proposed in a linear contextual bandit problem.
problem Selecting candidates from different sensitive groups in a fair manner.
method A greedy policy that constructs a ridge regression estimate and computes relative rank using empirical cumulative distribution function.
result The greedy policy achieves fair pseudo-regret of order d T \sqrt{dT} d T after T T T rounds, satisfying demographic parity. This paper formalizes reward training in RLHF and provides theoretical guarantees.
problem Costly human feedback collection in RLHF.
method Linear contextual dueling bandit method for regret minimization.
result Derives bounds on simple regret for offline reward training.
Improved confidence bounds for linear logistic model with applications to bandits.
problem Improving confidence bounds for linear logistic model.
method Self-concordant analysis of the logistic loss to avoid dependence on worst-case variance.
result Significant improvement in confidence bounds, avoiding dependence on 1 / κ 1/κ 1/ κ .