A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Reinforcement learning with trajectory feedback instead of state-action rewards.
A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.
Trajectory-level supervision allows efficient offline reinforcement learning.
Robots need models of human behavior for both inferring human goals and preferences, and predicting what people will do. A common model is the Boltzmann noisily-rational decision model, which assumes people approximately optimize a reward function and choose trajectories in proportion to their exponentiated reward. Whi…
PlanGAN uses GANs to plan efficient trajectories for multi-goal tasks in sparse reward environments.
Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from…
OSIL learns safe policies from unsafe demonstrations.
This paper proposes a new algorithm for learning guidance rewards in RL.
This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproduc…
Reinforcement learning with sparse rewards is challenging because an agent can rarely obtain non-zero rewards and hence, gradient-based optimization of parameterized policies can be incremental and slow. Recent work demonstrated that using a memory buffer of previous successful trajectories can result in more effective…
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
Recent advances in deep reinforcement learning algorithms have shown great potential and success for solving many challenging real-world problems, including Go game and robotic applications. Usually, these algorithms need a carefully designed reward function to guide training in each time step. However, in real world, …
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
New algorithm infers reward function from agent's learning trajectories.
Unified LP framework for offline reward learning from human demonstrations and feedback.
New RL formulation for maximizing maximum reward in molecule generation.
SafeMIL learns safer policies by avoiding risky behavior from non-preferred trajectories.
We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often …
Unified framework improves diffusion model rewards without full trajectories.
TLRS improves predictive power of mined formulaic alpha factors.
Boosted GFlowNets improve exploration by sequentially training GFlowNets with residual rewards.
Modeling driver trajectories using inverse reinforcement learning and random utility.
Paper tackles offline preference-based RL with human feedback.
Enhanced decision-making through Dreamer's anticipatory trajectories and Online Decision Transformer.
Learning to solve complex goal-oriented tasks with sparse terminal-only rewards often requires an enormous number of samples. In such cases, using a set of expert trajectories could help to learn faster. However, Imitation Learning (IL) via supervised pre-training with these trajectories may not perform as well and gen…
Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However…
Develops a new model to predict training dynamics of large language models.
A new reinforcement learning method reduces action complexity for robust control.
We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate. MAPO is applicable to deterministic environments with discrete actions, such as structured prediction and combinatorial optimization ta…
Proposes a new theoretical framework for PbRL that requires less human feedback.
Reinforcement learning offers the promise of automating the acquisition of complex behavioral skills. However, compared to commonly used and well-understood supervised learning methods, reinforcement learning algorithms can be brittle, difficult to use and tune, and sensitive to seemingly innocuous implementation decis…
Unified framework for human-like decision making in various sequential tasks.
Study shows fast rates for inverse reinforcement learning with linear rewards.
Enhances reinforcement learning from sparse data.
Deep Reinforcement Learning has shown tremendous success in solving several games and tasks in robotics. However, unlike humans, it generally requires a lot of training instances. Trajectories imitating to solve the task at hand can help to increase sample-efficiency of deep RL methods. In this paper, we present a simp…
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. …
Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial condi…
This work characterizes reward function partial identifiability and its impact on policy optimization.
We present a method for learning options from segmented demonstration trajectories. The trajectories are first segmented into skills using nonparametric Bayesian clustering and a reward function for each segment is then learned using inverse reinforcement learning. From this, a set of inferred trajectories for the demo…
Develops HMRL for sparse reward RL problems, improving meta policy efficiency and transferability.
Task-agnostic RL tackles exploration in MDPs with multiple tasks.
We consider the problem of learning from demonstrated trajectories with inverse reinforcement learning (IRL). Motivated by a limitation of the classical maximum entropy model in capturing the structure of the network of states, we propose an IRL model based on a generalized version of the causal entropy maximization pr…
In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…
Paper introduces OTR for efficient offline RL in surgical robotics.
Meta-algorithm for efficient reinforcement learning from human preferences.
New method boosts skill learning by encouraging optimistic exploration.
A key impediment to reinforcement learning (RL) in real applications with limited, batch data is defining a reward function that reflects what we implicitly know about reasonable behaviour for a task and allows for robust off-policy evaluation. In this work, we develop a method to identify an admissible set of reward f…
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.