PQR estimates reward functions from actions and states without assuming state-only rewards.
problem Estimating reward functions from actions and states without state-only assumptions.
method Deep learning approach that sequentially estimates policy, Q-function, and reward.
result PQR uniquely recovers true reward with known transitions and bounds error with unknown transitions.
Unified framework for estimating reward functions in competitive games.
problem Estimating unknown reward functions in competitive games.
method Unified framework with entropy regularization for reward function recovery.
result Strong theoretical guarantees and practical effectiveness demonstrated.
Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the rew…
Paper tackles robust offline RL with heavy-tailed rewards.
problem Real-world applications often encounter heavy-tailed rewards, challenging offline RL.
method Proposes ROAM and ROOM algorithms using median-of-means method for robust off-policy evaluation and OPO.
result Demonstrates superior performance on heavy-tailed reward datasets compared to existing methods.
A new framework estimates expert policy support to create a reward function for imitation learning.
problem Imitation learning from expert trajectories without reinforcement signals.
method Estimating the support of the expert policy to compute a fixed reward function.
result Comparable or better performance than state-of-the-art methods on discrete and continuous domains.
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
NeuralRBMLE tackles explore-exploit trade-offs in contextual bandits with neural networks.
problem Stochastic contextual bandit problem with general bounded reward functions.
method Reward-biased maximum likelihood estimation with neural networks to enforce exploration.
result Both NeuralRBMLE variants achieve O ~ ( T ) \widetilde{\mathcal{O}}(\sqrt{T}) O ( T ) regret. New algorithm estimates treatment effects for more efficient contextual bandits.
problem Contextual bandits struggle with action-independent reward redundancies.
method Reduces contextual bandits to heterogeneous treatment effect estimation.
result Heterogeneous treatment effect estimation leads to more efficient model estimation.
A new method recovers rewards from behavior policies using classification and regression.
problem Recovering meaningful rewards from observed behavior in reinforcement learning.
method GenPQR, a modular procedure that estimates behavior policy, evaluates soft Q-function, and recovers normalized reward using classification and regression.
result GenPQR matches or improves reward recovery compared to DeepPQR, while being simpler and more modular.
New algorithms improve efficiency in learning from personalized rewards.
problem Learning from personalized rewards in recommendation systems.
method Developed provably efficient algorithms with sublinear regret for context-dependent feedback.
result Introduced a Lipschitz reward estimator that improves generalization performance.
SIRL recovers reward function probability distribution from expert actions.
problem Recovering reward functions from expert demonstrations in reinforcement learning.
method Monte Carlo Expectation-Maximization (MCEM) method to estimate reward function probability distribution.
result SIRL provides a robust and transferable solution to the IRL problem.
Unified meta algorithms estimate various distribution functionals in infinite-armed bandits.
problem Estimating various distribution functionals in infinite-armed bandits.
method Unified meta algorithms for offline and online settings, achieving optimal sample complexities.
result Online estimation offers significant advantage for certain distribution functionals.
Unified LP framework for offline reward learning from human demonstrations and feedback.
problem Reward learning from human demonstrations and feedback with robustness and sample efficiency.
method A novel linear programming framework for offline reward learning.
result Unified LP framework achieves better performance compared to MLE.
Paper addresses reward learning issues in RL, improving both under- and over-estimation.
problem Reward learning from data can lead to reward delusions or underestimation, causing unintended behaviors.
method Connects reward learning to positive-unlabeled (PU) learning and applies a large-scale PU learning algorithm.
result Improves both GAIL and supervised reward learning without additional assumptions.
Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.
problem Non-uniqueness of reward functions in inverse reinforcement learning.
method Entropy regularization combined with least-squares reconstruction of the reward from the soft Bellman residual.
result Least-squares reward function is unique and consistent with the expert policy.
Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However…
New approach for reward-free exploration reduces estimation error.
problem Reward-free exploration in reinforcement learning.
method Adaptive approach reducing MDP estimation error.
result Reward-free UCRL algorithm improves sample complexity.
MFMs enable efficient reward alignment for generative models.
problem Computational bottleneck in controlling generative models.
method Meta Flow Maps (MFMs) extend consistency models and flow maps to stochastic regime for efficient value function estimation.
result MFMs enable inference-time steering and unbiased, off-policy fine-tuning to general rewards efficiently.
Paper introduces a novel reward function for noisy financial markets using imitation learning.
problem Noisy reward function in financial markets hinders RL agent performance.
method Integrates imitation learning feedback with reinforcement learning to improve reward function design.
result Improves financial performance metrics compared to traditional benchmarks and RL agents.
B-REX efficiently learns Atari game policies from pixel inputs using Bayesian methods.
problem Learning reward functions from visual inputs with uncertainty and safety considerations.
method Bayesian Reward Extrapolation (B-REX) using successor features and preferences.
result B-REX generates posterior samples efficiently, enabling high-confidence performance bounds.
Inverse Reinforcement Learning (IRL) describes the problem of learning an unknown reward function of a Markov Decision Process (MDP) from observed behavior of an agent. Since the agent's behavior originates in its policy and MDP policies depend on both the stochastic system dynamics as well as the reward function, the …
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
Direct approach for handling contextual bandits with latent state dynamics.
problem Handling contextual bandits with latent state dynamics, especially when rewards depend on posterior probabilities of hidden states.
method Direct reduction to standard linear contextual bandits, extended analysis of HMM parameters, periodic update of reward-model parameters.
result Periodic update of reward-model parameters allows handling complex dependencies in hidden states.
Study ℓ ∞ \ell_\infty ℓ ∞ bounds for MRP value function estimation.
problem Estimate MRP value function from samples.
method Analyze standard and robust plug-in approaches, establish bounds.
result Non-asymptotic and data-dependent ℓ ∞ \ell_\infty ℓ ∞ -norm bounds. Bayesian REX learns Atari games from demonstrations efficiently.
problem Bayesian reward learning for complex control problems is computationally intractable.
method Bayesian Reward Extrapolation (Bayesian REX) pre-trains a low-dimensional feature encoding and uses preferences to perform fast Bayesian inference.
result Bayesian REX learns Atari games from demonstrations in 5 minutes, competitive with state-of-the-art methods.
New approach transfers rewards learned in one environment to reinforcement learning in a new environment.
problem Transfer of rewards learned using inverse reinforcement learning from one environment to a new, different environment.
method Formulate the problem as a joint system of Bellman equations, develop minimax estimators for the target soft- q q q -function, solve the source and target system of equations jointly. result The coupled approach removes the first-order influence of source Bellman residual error compared to the sequential approach.
A new framework for multi-agent inverse reinforcement learning.
problem Finding suitable reward functions for multi-agent scenarios.
method MA-AIRL, based on a new solution concept and maximum pseudolikelihood estimation.
result MA-AIRL can recover highly correlated reward functions and significantly outperforms prior methods.
Off-policy evaluation for MNAR rewards in MDPs
problem Off-policy evaluation in MDPs with MNAR rewards
method Formalizing a reward-dependent propensity model and using future states as shadow variables
result Proposed an Fitted-Q-Evaluation-style estimator that propagates recovered rewards while allowing target policies to depend on past missingness indicators
New method optimizes policies without assuming known link functions between preferences and rewards.
problem Policy alignment with unknown and unrestricted link functions.
method Formulates an f f f -divergence-constrained reward maximization problem, learning policies directly. result Induces a semiparametric single-index binary choice model for policy alignment.
Improved regret for single-index bandits with optimal algorithm.
problem Optimizing rewards from unknown one-dimensional projections of high-dimensional contexts.
method Two-phase algorithm: first estimate projection direction, then discretize and use UCB.
result Achieved optimal regret of i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) . The paper develops a robust algorithm for contextual bandits with heavy-tailed rewards.
problem Contextual bandits with heavy-tailed rewards.
method Develops an algorithm based on Catoni's estimator for robust statistics, applying it to contextual bandits with general function approximation.
result Establishes regret bounds that depend on cumulative reward variance and logarithmically on the reward range and number of rounds.
New algorithm learns optimal policies without explicit rewards.
problem Learning optimal policies without explicit rewards.
method Developed a computationally tractable algorithm for reward-free navigation using linear value iteration.
result PAC guarantees on learning near optimal value functions and policies.
Paper proposes Langevin dynamics for adaptive IRL of stochastic gradient algorithms.
problem Estimating reward functions from noisy gradient estimates of stochastic gradient agents.
method Generalized Langevin dynamics algorithm for IRL.
result Proposed algorithms asymptotically generate samples proportional to exp(R(θ)).
New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of O ( d 2 ε − 2 ) O(d^2\varepsilon^{-2}) O ( d 2 ε − 2 ) for finding an ε \varepsilon ε -optimal policy. GFlowNets sample diverse candidates in active learning.
problem Sampling diverse candidates in active learning.
method Generative Flow Networks (GFlowNets) for proportional sampling.
result GFlowNets estimate joint and marginal distributions.
We decode latent states in Block MDPs and learn near-optimal policies.
problem Model estimation and reward-free learning in Block MDPs.
method Information-theoretical lower bound and efficient model estimation algorithm.
result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.
Paper tackles offline preference-based RL with human feedback.
problem Offline Preference-based Reinforcement Learning with preference feedback.
method Two-step approach: MLE for reward estimation and distributionally robust planning.
result First guarantee for learning any target policy with polynomial samples.
A new method resolves non-identifiability in reward modeling using anchor labels.
problem Non-identifiability in reward modeling from pairwise preferences alone.
method Anchor-guided Variance-aware Reward Modeling (AVRM) framework.
result AVRM resolves non-identifiability and improves reward modeling performance.
Learning an optimal policy from a multi-modal reward function is a challenging problem in reinforcement learning (RL). Hierarchical RL (HRL) tackles this problem by learning a hierarchical policy, where multiple option policies are in charge of different strategies corresponding to modes of a reward function and a gati…
The paper proposes a method to decompose value functions in RL for better understanding and prediction.
problem Understanding and predicting the dynamics and returns in reinforcement learning models.
method A two-step approach decomposing the value function into future dynamics and trajectory returns, with a practical deep RL algorithm.
result The proposed algorithm outperforms in MuJoCo tasks, especially under delayed reward settings.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Improved Q-learning with hindsight factor reduces overestimation errors.
problem Overestimation errors in Q-learning leading to suboptimal policies.
method Introducing a hindsight factor as an additional loss term in Q-learning.
result The hindsight factor significantly reduces overestimation errors and improves stability.
Estimates rewards from a demonstrator's learning process.
problem Estimating rewards from a demonstrator's behavior.
method Leveraging the demonstrator's exploration phase for reward estimation.
result Consistent reward estimation possible without identifiability issues.
Paper develops robust estimators and strategies for stochastic MABs with heavy-tailed rewards.
problem Stochastic multi-armed bandits with heavy-tailed rewards.
method Proposes a novel robust estimator and perturbation-based exploration strategy.
result Develops upper and lower regret bounds for various perturbations.
New bandit algorithm detects and adapts to seasonal changes in rewards.
problem Adapting to abrupt changes in user preferences during events.
method Detects and adapts to seasonal changes in reward function.
result Outperforms state-of-the-art algorithms for non-stationary environments.
New algorithm reduces regret for single-index bandits to nearly optimal.
problem Optimizing rewards from unknown projections of high-dimensional contexts.
method Two-phase algorithm: estimate projection direction, reduce to 1D bandit, use UCB.
result Achieved optimal regret of i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) . Algorithm estimates human decision-making in high-dimensional states with finite-time guarantees.
problem Estimating optimal policies and measures of fit in dynamic decision models with high-dimensional state spaces.
method Single-loop estimation algorithm with stochastic gradient steps for reward maximization.
result Algorithm converges to a stationary solution with finite-time guarantees and approximates maximum likelihood sublinearly.
Novel algorithm reduces computational burden in IRL with finite-time guarantees.
problem Efficiently recover reward function and optimal policy from expert behavior.
method Single-loop algorithm that maximizes likelihood after each policy improvement step.
result Algorithm provably converges to a stationary solution with finite-time guarantees.