Proposes a method to boost deep reinforcement learning with sparse rewards.
problem Challenges in learning complex behaviors with long horizons and sparse rewards.
method Predictive coding for reward shaping.
result Achieves better learning by providing reward signals that understand environment dynamics and emphasize useful features.
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.
Method predicts future rewards from past actions in a linear Gaussian system.
problem Maximizing cumulative reward in a stochastic multi-armed bandit with linear Gaussian dynamics.
method Proposes a method using a modified Kalman filter to predict future rewards based on past rewards.
result Reward from any action can be used to predict another action's future reward.
Research predicts cryptocurrency staking rewards with high accuracy.
problem Predicting cryptocurrency staking rewards.
method Two predictive methodologies: sliding-window average and linear regression models.
result ETH staking rewards can be forecasted with RMSE within 0.7% and 1.1% of the mean value for 1-day and 7-day look-aheads respectively.
Proposes learning latent reward model for planning from rewards.
problem Planning in high-dimensional state spaces with limited reward information.
method Directly learns a latent dynamics model from rewards, planning in latent state-space.
result Successfully learns accurate latent reward prediction model, achieving strong performance and high sample efficiency.
New method uses reward prediction error for efficient exploration.
problem Efficient exploration in reinforcement learning, especially in complex environments.
method Reward prediction error (RPE) as an intrinsic motivation for exploration, combined with a deep reinforcement learning method (QXplore).
result QXplore outperforms state-novelty methods in diverse tasks, especially when state novelty is not correlated with improved reward.
Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrins…
This study proposes hidden state curiosity to enhance RL models' resilience against noise.
problem Curiosity traps in RL models distract agents from discovering novel experiences.
method Proposed hidden state curiosity based on the Free Energy Principle to reward agents for KL divergence between predictive priors and posteriors.
result Agents with hidden state curiosity are more resilient against curiosity traps compared to those with prediction error curiosity.
Curiosity-Critic improves world model training by focusing on cumulative prediction error.
problem Training world models with intrinsic rewards that consider cumulative prediction error.
method Curiosity-Critic uses a surrogate reward based on the difference between current and asymptotic prediction errors, estimated online by a co-trained critic.
result Curiosity-Critic outperforms other methods in training speed and final world model accuracy.
New method predicts and optimizes test-time scaling for LLMs.
problem Lack of principled guidance on scaling LLMs efficiently.
method Tail-guided search to predict and allocate compute.
result SLG Search achieves higher rewards with less compute.
TLRS improves predictive power of mined formulaic alpha factors.
problem Sparse rewards in RL for mining formulaic alpha factors.
method Trajectory-level Reward Shaping (TLRS) with reward centering.
result TLRS boosts predictive power by 9.29% over existing methods.
New algorithm uses machine learning to predict rewards for decision-making problems.
problem Sequential decision-making under uncertainty with scarce online data.
method Machine Learning-Assisted Upper Confidence Bound (MLA-UCB) algorithm.
result Proves to improve cumulative regret even with biased surrogate rewards.
The Artificial Prediction Market is a recent machine learning technique for multi-class classification, inspired from the financial markets. It involves a number of trained market participants that bet on the possible outcomes and are rewarded if they predict correctly. This paper generalizes the scope of the Artificia…
In structured output prediction tasks, labeling ground-truth training output is often expensive. However, for many tasks, even when the true output is unknown, we can evaluate predictions using a scalar reward function, which may be easily assembled from human knowledge or non-differentiable pipelines. But searching th…
Reinforcement learning is concerned with identifying reward-maximizing behaviour policies in environments that are initially unknown. State-of-the-art reinforcement learning approaches, such as deep Q-networks, are model-free and learn to act effectively across a wide range of environments such as Atari games, but requ…
This work presents a game-theoretic method for AVs that handles imperfect communication and individual rewards.
problem Real-time, imperfect communication and individual rewards in multi-agent interactions.
method Game-theoretic approach that allows for imperfect communication and individual rewards.
result More realistic assumptions lead to better reward inference and prediction of future actions.
This paper explores intrinsic rewards to improve learning from multiple value functions.
problem How to adapt reinforcement learning systems to optimize learning from multiple value functions.
method Investigated and compared 14 different intrinsic reward mechanisms in a new bandit-like parallel-learning testbed.
result Intrinsic rewards based on the amount of learning can generate useful behavior, if each individual learner is introspective.
Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However…
Improves reward bounds for prediction with expert advice using abstention.
problem Prediction with expert advice under bandit feedback with abstention.
method CBA algorithm exploiting abstention to improve reward bounds.
result Achieved significant improvement in reward bounds for general confidence-rated predictors.
Paper simplifies complex AI exploration by predicting future rewards.
problem Training machines to optimally gather complex information.
method Developed a denser reward structure using cross-value to decouple exploration and exploitation.
result Demonstrated successful learning of challenging tasks without shaping or bonuses.
Learning in sparse reward settings remains a challenge in Reinforcement Learning, which is often addressed by using intrinsic rewards. One promising strategy is inspired by human curiosity, requiring the agent to learn to predict the future. In this paper a curiosity-driven agent is extended to use these predictions di…
PFN-TS uses Thompson sampling with PFNs to improve contextual bandit performance.
problem Improving contextual bandit performance using Thompson sampling with prior-data fitted networks.
method PFN-TS converts PFN posterior predictives into mean-reward samples using a subsampled predictive central limit theorem.
result PFN-TS achieves the best average rank across nonlinear synthetic and OpenML classification-to-bandit benchmarks.
This work improves MARL exploration efficiency using predictive guidance and prioritized experience replay.
problem Challenges in multi-agent reinforcement learning, especially in exploration efficiency and reward informativeness.
method A predictive network forecasts reward, guiding MARL to accelerate exploration. Improved prioritized experience replay utilizes TD error effectively.
result The proposed algorithm outperforms existing methods in cooperative multi-agent environments.
Develops a new model to predict training dynamics of large language models.
problem Lack of mechanistic understanding of training dynamics in large language models.
method A first-principles reduced-order model of training dynamics, predicting group-size invariance and stability thresholds.
result Closed-form model predicts training dynamics with high accuracy and provides new diagnostics.
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. …
A new method resolves non-identifiability in reward modeling using anchor labels.
problem Non-identifiability in reward modeling from pairwise preferences alone.
method Anchor-guided Variance-aware Reward Modeling (AVRM) framework.
result AVRM resolves non-identifiability and improves reward modeling performance.
New predictive bandit model with noise for better decision making.
problem Optimizing decisions with limited information and noise.
method Introduced predictive bandits, derived regret bounds, and developed algorithms.
result Developed algorithms matching theoretical regret bounds.
The paper explores a Multi-Objective RL approach for trading that generalizes reward functions.
problem Improving performance in single-asset trading through adaptive reward functions.
method Developed a Multi-Objective Deep Reinforcement Learning algorithm to generalize reward functions and discount factors.
result The Multi-Objective algorithm demonstrates increased predictive stability and better performance in sparse reward scenarios.
New method combines curiosity and hindsight for stacking blocks.
problem Sparse rewards in reinforcement learning.
method Curiosity-driven exploration combined with hindsight and curriculum learning.
result First to stack more than two blocks using only sparse reward.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
The paper proposes a method to decompose value functions in RL for better understanding and prediction.
problem Understanding and predicting the dynamics and returns in reinforcement learning models.
method A two-step approach decomposing the value function into future dynamics and trajectory returns, with a practical deep RL algorithm.
result The proposed algorithm outperforms in MuJoCo tasks, especially under delayed reward settings.
EOI enhances individuality in multi-agent systems.
problem Improving multi-agent cooperation through individuality.
method A probabilistic classifier that predicts agent identity based on observations, with intrinsic rewards and regularizers.
result EOI significantly outperforms existing methods in multi-agent cooperative scenarios.
EPIC quantifies reward differences without policy optimization.
problem Distinguishing reward function quality from policy optimization issues.
method EPIC distance to compare reward functions directly.
result EPIC bounds policy training success and regret.
Study scaling laws of reward model overoptimization in reinforcement learning.
problem Reward model overoptimization hinders true performance in reinforcement learning.
method Synthetic setup with fixed gold reward model; optimization using RL or best-of-n sampling; analysis of scaling laws. result Scaling laws of reward model overoptimization differ based on optimization method and scale smoothly with model parameters.
Agents cooperate to make decisions in multi-armed bandits over a graph.
problem Optimizing decisions in multi-agent multi-armed bandits with shared information.
method Designs consensus-based distributed estimation and cooperative algorithms for group decision-making.
result Achieves group performance close to centralized fusion center.
Algorithm learns user's reward function from hypothetical behaviors.
problem Aligning agent behavior with unknown user objectives.
method Synthesizes hypothetical behaviors, asks user for rewards, trains neural network.
result Significantly outperforms prior methods in learning reward models.
This paper interpolates reward functions to predict optimal value functions in MORL.
problem Finding optimal value functions in MORL requires recomputing for each set of weights.
method Interpolating reward function weights to smooth value function transformations.
result Smooth interpolation of optimal value functions over reward function weights.
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agen…
Kernel-based function approximation improves reinforcement learning performance.
problem Average reward reinforcement learning in infinite horizon settings.
method Optimistic algorithm based on kernel ridge regression.
result No-regret performance guarantees and confidence intervals for kernel-based predictions.
AutoDIME automates design of multi-agent environments for RL.
problem Designing multi-agent environments for reinforcement learning is challenging.
method Developed intrinsic teacher rewards for multi-agent settings and evaluated them in various tasks.
result Value disagreement was found to be most consistent and effective across tasks.
Imitation by observation is an approach for learning from expert demonstrations that lack action information, such as videos. Recent approaches to this problem can be placed into two broad categories: training dynamics models that aim to predict the actions taken between states, and learning rewards or features for com…
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
Improved text generation using transferable rewards from related tasks.
problem Non-differentiable task-specific scores limit the use of policy gradient methods in text generation.
method Transferable Reward Learner that uses model-based rewards for sentence-level and phrase-level similarity.
result Improved performance on semantic evaluation measures in image captioning tasks.
A network of spiking agents learns complex tasks using global reward signals.
problem Solving complex reinforcement learning tasks.
method A hierarchical network of GLM spiking agents, each modulating its firing policy based on local and global reward signals.
result A network of spiking agents can learn complex action representations to solve RL tasks.
Machine learning is often used in competitive scenarios: Participants learn and fit static models, and those models compete in a shared platform. The common assumption is that in order to win a competition one has to have the best predictive model, i.e., the model with the smallest out-sample error. Is that necessarily…
Reinforcement learning algorithms struggle when the reward signal is very sparse. In these cases, naive random exploration methods essentially rely on a random walk to stumble onto a rewarding state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or …
The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.
problem Reward modeling in policy optimization and its impact on downstream value.
method Two-stage neural reward model: first learns hidden direction, then fits readout layer.
result For any feature-learning temperature above a dimension-free threshold, a constant fraction of neurons recover the hidden direction.
STAR framework reduces OPE variance by distilling complex problems into discrete ARPs.
problem High variance and bias in off-policy evaluation methods.
method STAR framework that includes various OPE estimators and leverages state abstraction.
result Predictions from ARPs estimated from off-policy data are asymptotically correct.