arXiv research

Direct approach for handling contextual bandits with latent state dynamics.

SWIFT learns intrinsic rewards from LLM hidden states for efficient best-of-N sampling.

This study proposes hidden state curiosity to enhance RL models' resilience against noise.

RL agents learn from a few tasks to generalize to new ones.

This paper addresses reward estimation and incentive design for agents with hidden rewards.

The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.

In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in a…

Paper tackles hidden game problem in AI alignment and language games.

Modeling decision-making dynamics in MDD patients using RL-HMM.

Study of multi-armed bandits with state-switching rewards using Markov models.

We consider a finite-armed structured bandit problem in which mean rewards of different arms are known functions of a common hidden parameter $θ^*$. Since we do not place any restrictions of these functions, the problem setting subsumes several previously studied frameworks that assume linear or invertible reward funct…

Proposes a new theoretical framework for PbRL that requires less human feedback.

Algorithm improves RL by discovering delayed causal relations.

Language models perform worse with implicit reward models than explicit ones.

Bayesian optimization improved for biased data.

Study symmetric linear bandits with hidden symmetry, achieving improved regret bounds.

New method for RL with general utilities using variational policy gradient.

We consider the stochastic linear (multi-armed) contextual bandit problem with the possibility of hidden simple multi-armed bandit structure in which the rewards are independent of the contextual information. Algorithms that are designed solely for one of the regimes are known to be sub-optimal for the alternate regime…

We consider the problem of learning to behave optimally in a Markov Decision Process when a reward function is not specified, but instead we have access to a set of demonstrators of varying performance. We assume the demonstrators are classified into one of k ranks, and use ideas from ordinal regression to find a rewar…

Paper analyzes finite-time guarantees for preference-based RL.

A RL framework selects features to balance bias and accuracy dynamically.

We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a cu…

Recent work has identified that classification models implemented as neural networks are vulnerable to data-poisoning and Trojan attacks at training time. In this work, we show that these training-time vulnerabilities extend to deep reinforcement learning (DRL) agents and can be exploited by an adversary with access to…

LaTRO optimizes latent reasoning in LLMs without external reward.

Temporal point process is an expressive tool for modeling event sequences over time. In this paper, we take a reinforcement learning view whereby the observed sequences are assumed to be generated from a mixture of latent policies. The purpose is to cluster the sequences with different temporal patterns into the underl…

Study on multi-agent decision making complexity, showing sample efficiency gaps.

We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…

We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…

This paper introduces a new approach to finding knots and links with hidden symmetries using "hidden extensions", a class of hidden symmetries defined here. We exhibit a family of tangle complements in the ball whose boundaries have symmetries with hidden extensions, then we further extend these to hidden symmetries of…

JEPA fails to improve language model performance when fine-tuning.

A new algorithm learns from raw feature vectors using deep neural networks and UCB for exploration.

Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.

Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.

This work analyzes the value of future reward information in RL.

We propose and study a new model for reinforcement learning with rich observations, generalizing contextual bandits to sequential decision making. These models require an agent to take actions based on observations (features) with the goal of achieving long-term performance competitive with a large set of policies. To …

Self-supervised reward prediction improves RL in sparse reward settings.

The study categorizes reward errors in reinforcement learning, finding some can be beneficial.

Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.

Reward models need more than just accuracy for effective RLHF.

Proposes a method to boost deep reinforcement learning with sparse rewards.

Action guidance helps agents learn true objectives in games with sparse rewards.

We introduce a stochastic contextual bandit model where at each time step the environment chooses a distribution over a context set and samples the context from this distribution. The learner observes only the context distribution while the exact context realization remains hidden. This allows for a broad range of appl…

New RL method uses distance between states instead of rewards for sparse reward environments.

Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond …

Enhances reward specification in RL with a novel language-based approach.

Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.

We propose a generic, Bayesian, information geometric approach to the exploration--exploitation trade-off in multi-armed bandit problems. Our approach, BelMan, uniformly supports pure exploration, exploration--exploitation, and two-phase bandit problems. The knowledge on bandit arms and their reward distributions is su…