The paper tackles control policy learning for unknown systems using convex optimization.
problem Learning control policies for unknown linear dynamical systems to maximize a quadratic reward function.
method Sequential convex programming to optimize expected reward over posterior system parameter distribution.
result The method achieves reliable local convergence and robust stability, demonstrated with strong performance and robustness in simulations and real-world applications.
Study derivative-free methods for linear policies in linear-quadratic systems.
problem Optimizing policies in linear-quadratic systems with limited derivative information.
method Derivative-free methods applied to linear policies over various noise and reward feedback settings.
result These methods converge to near-optimal policies with a polynomial number of zero-order evaluations.
Study shows a linear quadratic regulator's imitation learning converges globally.
problem Global convergence of imitation learning for linear quadratic regulators.
method Analyzed alternating gradient algorithm and established Q-linear rate of convergence.
result Established a unique saddle point for globally optimal policy and reward function.
A new method eliminates reward estimation variance in sequential decision processes.
problem High variance in gradient estimation hinders sample efficiency in reinforcement learning.
method Proposes an unbiased method that completely eliminates variance under certain conditions.
result The proposed method significantly improves performance in challenging problems with delayed rewards.
Improves generative models by optimizing rewards and sample editing.
problem Efficiently generating high-reward samples with structural constraints.
method Introduces MDM-VGB, a discrete diffusion sampler that augments unmasking generation with reward-guided remasking.
result MDM-VGB achieves quadratic complexity and robustness to noise, outperforming heuristics like best-of- N N N . HTRPO tackles sparse rewards in RL with improved stability and performance.
problem Sparse rewards in reinforcement learning.
method HTRPO extends TRPO with hindsight and QKL for better policy update stability.
result HTRPO consistently outperforms TRPO and HPG in various sparse reward tasks.
Improves reward bounds for prediction with expert advice using abstention.
problem Prediction with expert advice under bandit feedback with abstention.
method CBA algorithm exploiting abstention to improve reward bounds.
result Achieved significant improvement in reward bounds for general confidence-rated predictors.
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) regret. Quantum algorithms for multi-armed bandits are explored with limited reward access.
problem Exploring quantum speed-ups in multi-armed bandit problems with limited reward information.
method Introduced new bandit models and showed query complexity equivalence with classical algorithms.
result No quadratic speed-up is possible for multi-armed bandits with limited reward access.
Reinforcement learning for optimizing retirement plans and target dated funds.
problem Optimizing financial goals through periodic investments and withdrawals.
method G-Learner and GIRL algorithms for goal-based wealth management.
result G-Learner provides a computationally tractable solution for wealth management tasks.
New method learns Atari game Montezuma's Revenge from a single demonstration.
problem Learning from sparse rewards in complex exploration tasks.
method Maximizing rewards directly from a single demonstration state, combined with off-the-shelf reinforcement learning.
result Trained agent achieves high-score of 74,500 in Montezuma's Revenge.
Study aims to optimize financial investments by balancing risk and reward efficiently.
problem Balancing risk and reward in dynamic financial investments.
method Proposes a reinforcement learning method to maximize expected quadratic utility, focusing on first and second moments of rewards.
result The proposed method yields MV-efficient policies that maximize expected reward without increasing variance.
New algorithms predict reinforcement learning values efficiently.
problem Predicting reinforcement learning values with linear function approximation.
method Multi-timescale stochastic approximation of cross entropy method.
result Proved convergence and achieved good performance in experiments.
New model-free algorithm for LQ control with sublinear regret.
problem Model-free reinforcement learning for adaptive control of linear quadratic systems.
method Reduction to expert prediction problem, policy iteration with forced exploration.
result Algorithm achieves sublinear regret and polynomial computation cost.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
New algorithms learn MDPs with continuous states and actions using Gaussian processes.
problem Online learning in unknown, episodic MDPs with continuous states and actions.
method Developed variants of UCRL and posterior sampling algorithms using Gaussian process priors.
result Sublinear regret bounds for learning MDPs with specific kernel structures.
Reinforcement learning for continuous-time risk-sensitive asset allocation
problem Continuous-time risk-sensitive asset allocation
method Free energy-entropy duality reformulation and q q q -learning actor-critic method result Optimal policy learning with high accuracy
New L2 regularization improves softmax MAB performance.
problem Improving softmax MAB performance with vanishing regularization.
method L2 regularization with vanishing parameter analyzed and proven convergent.
result Vanishing L2 regularization makes softmax MAB more numerically advantageous.
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
New algorithm reduces suboptimality in imitation learning to nearly optimal levels.
problem Statistical limits of imitation learning in MDPs with known transitions.
method Mimic-MD algorithm and reduction to value estimation problem.
result Upper bound of O ( ∣ S ∣ H 3 / 2 / N ) O(|\mathcal{S}|H^{3/2}/N) O ( ∣ S ∣ H 3/2 / N ) for suboptimality, with efficient computation. New algorithm catches moving subspaces in bandit problems.
problem Adapt to changing low-dimensional latent subspaces in bandit settings.
method Piecewise-stationary low-rank linear contextual bandits with CUSUM-style boundary detection.
result Achieves intrinsic rank dynamic regret rate of O ( r T ) O(r\sqrt{T}) O ( r T ) . Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.
Paper addresses reward learning issues in RL, improving both under- and over-estimation.
problem Reward learning from data can lead to reward delusions or underestimation, causing unintended behaviors.
method Connects reward learning to positive-unlabeled (PU) learning and applies a large-scale PU learning algorithm.
result Improves both GAIL and supervised reward learning without additional assumptions.
This work analyzes the value of future reward information in RL.
problem Analyzing the impact of knowing future rewards in reinforcement learning.
method Competitive analysis and worst-case reward distribution.
result Exact ratios between standard RL agents and those with future-reward lookahead.
Method learns reward functions that are independently obtainable and sum to original reward.
problem Learning reward functions that are independent and meaningful.
method Defining independent obtainability and optimizing a novel objective function.
result Learned reward functions generalize well to modified environments and have optimal policies.
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.
problem Reward collapse in aligning large language models with human preferences.
method Introduced a prompt-aware optimization scheme to derive closed-form expressions for reward distributions.
result Our prompt-aware utility functions significantly alleviate reward collapse during training.
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.
Direct contextual policy search methods learn to improve policy parameters and simultaneously generalize these parameters to different context or task variables. However, learning from high-dimensional context variables, such as camera images, is still a prominent problem in many real-world tasks. A naive application o…
Reward models need more than just accuracy for effective RLHF.
problem The effectiveness of reward models in RLHF is not fully understood.
method An optimization perspective to evaluate reward models.
result Reward models with low reward variance can lead to a flat optimization landscape, hindering performance.
Polynomial-time algorithms for identifying the best super arm in full-bandit feedback.
problem Finding the best super arm in a set of single arms with full-bandit feedback.
method Proposed polynomial-time bandit algorithms and an approximation algorithm for the 0-1 quadratic maximization problem.
result Polynomial-time algorithms for top-k selection problems.
Proposes a method to boost deep reinforcement learning with sparse rewards.
problem Challenges in learning complex behaviors with long horizons and sparse rewards.
method Predictive coding for reward shaping.
result Achieves better learning by providing reward signals that understand environment dynamics and emphasize useful features.
Reward estimation improves model-free RL performance under corrupted rewards.
problem Handling corrupted or stochastic rewards in reinforcement learning.
method Using an estimator for both rewards and value functions.
result Improves performance in various noise types and environments.
Extends Hindsight Experience Replay to learn from multiple reward functions.
problem Learning policies for multiple reward functions without trial runs.
method Develops a method to use a reward function to calculate rewards for states not encountered, and learns policies for multiple reward functions.
result A single policy can generalize across all linear combinations of multi-objective rewards.
Action guidance helps agents learn true objectives in games with sparse rewards.
problem Training agents in games with sparse rewards requires significant exploration.
method Action guidance, a novel technique that combines exploration with reward shaping.
result Action guidance enables agents to optimize true objectives efficiently.
New RL method uses distance between states instead of rewards for sparse reward environments.
problem Sparse rewards or non-reward environments in reinforcement learning.
method Uses goal-distance gradient and bridge point planning for policy improvement.
result Significantly better performance on sparse reward and local optimal problems in complex environments.
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
Enhances reward specification in RL with a novel language-based approach.
problem Reward specification in RL can lead to unintended, potentially harmful behaviours.
method Developed a novel class of language-based Reward Machines using RML's built-in memory.
result Can specify non-regular, non-Markovian reward functions for complex tasks.
Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.
problem Optimizing behavior for long-term goals in reinforcement learning with unstable long planning horizons.
method Reward tweaking learns a surrogate reward function that induces optimal behavior for the original task.
result Reward tweaking guides agents towards better long-term returns while planning for short horizons.
RUDDER simplifies Q-value estimation for delayed rewards in MDPs.
problem Solving delayed rewards in reinforcement learning with bias and variance issues.
method Reward redistribution and return decomposition to simplify Q-value estimation.
result RUDDER significantly speeds up Q-value estimation and improves performance on Atari games.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
BelMan uses Bayesian methods to optimize decisions in multi-armed bandit problems.
problem Optimizing decisions in multi-armed bandit problems with varying rewards and beliefs.
method BelMan uses a geometric approach with information projection and reverse projection to balance exploration and exploitation.
result BelMan outperforms other algorithms in specific scenarios involving many arms and continuous rewards.
Active Inverse Reward Design improves AI agent training by querying users for reward function preferences.
problem Iterative reward function tuning in AI agents is inefficient and may not generalize well.
method Structured queries to the user to compare reward functions, updating posterior with IRD.
result Substantially outperforms IRD in test environments, inferring non-linear rewards.
This work characterizes reward function partial identifiability and its impact on policy optimization.
problem Reward function partial identifiability in complex tasks.
method Formal characterisation of partial identifiability using various reward learning data sources.
result Unified framework for comparing data sources and downstream tasks by their invariances.
This paper introduces a new reward shaping method for average-reward reinforcement learning.
problem Speeding up convergence to an optimal policy in average-reward reinforcement learning tasks.
method Developed a temporal logic-based approach to automatically generate reward shaping functions.
result The optimal policy can be recovered using the proposed reward shaping framework.
Paper generalizes reward distribution in multi-armed bandits with temporally-partitioned rewards.
problem Handling partial rewards distributed over multiple rounds in multi-armed bandits.
method Introduces Beta-spread property to generalize reward distribution, derives lower bound, and provides TP-UCB-FR-G algorithm.
result Improves regret upper bound for some scenarios using Beta-spread property.
New reward function improves GAIL performance in task-based environments.
problem Reward bias in adversarial imitation learning.
method Proposed a new reward function to overcome existing biases.
result New reward function outperforms existing methods in task-based environments.