Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising way to train an agent to perform goal-directed tasks is to use Hindsight Learning approaches. In thes…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “hindsight experience”
Rewriting history improves RL algorithms for solving multiple tasks.
problem Improving sample efficiency in multi-task reinforcement learning.
method Introducing hindsight relabeling as inverse RL to generalize goal-relabeling techniques.
result Relabeling data using inverse RL accelerates learning in multi-task settings.
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…
HTRPO tackles sparse rewards in RL with improved stability and performance.
problem Sparse rewards in reinforcement learning.
method HTRPO extends TRPO with hindsight and QKL for better policy update stability.
result HTRPO consistently outperforms TRPO and HPG in various sparse reward tasks.
In Hindsight Experience Replay (HER), a reinforcement learning agent is trained by treating whatever it has achieved as virtual goals. However, in previous work, the experience was replayed at random, without considering which episode might be the most valuable for learning. In this paper, we develop an energy-based fr…
New method combines curiosity and hindsight for stacking blocks.
problem Sparse rewards in reinforcement learning.
method Curiosity-driven exploration combined with hindsight and curriculum learning.
result First to stack more than two blocks using only sparse reward.
HGG generates goals to improve sample efficiency in robotic tasks.
problem Efficiency in reinforcement learning with sparse reward signals.
method Generates valuable hindsight goals for reinforcement learning.
result Significantly improved sample efficiency over HER.
PCHID improves sample efficiency in reinforcement learning tasks.
problem Sparse rewards make learning policies difficult in reinforcement learning.
method Hindsight Inverse Dynamics with Hindsight Experience Replay and Policy Continuation.
result PCHID significantly improves sample efficiency and final performance on multi-goal tasks.
Framework uses hindsight regret to audit marketing budget allocations.
problem Lack of principled way to assess strategic budget allocations.
method Hindsight regret framework based on constraint-faithful benchmark.
result Identifies practical trade-off between allocation flexibility and detectability.
ACTRCE uses natural language to improve reinforcement learning performance.
problem Sparse reward in reinforcement learning.
method Extends HER framework with natural language goal representation.
result ACTRCE can solve challenging 3D navigation tasks and generalize to unseen instructions.
New method improves meta-reinforcement learning efficiency.
problem Sample inefficiency in meta-reinforcement learning.
method Hindsight Foresight Relabeling (HFR) method.
result HFR improves performance on various meta-reinforcement learning tasks.
New algorithm combines curriculum learning with HER for complex object manipulation tasks.
problem Learning complex sequential object manipulation tasks from scratch is challenging.
method Curriculum learning with Hindsight Experience Replay (HER) for recurrent object manipulation tasks.
result Significant improvement in learning sequential object manipulation tasks compared to vanilla-HER.
Interactive learning with hindsight instruction feedback achieves better performance than traditional methods.
problem Learning with expert supervision of optimal responses is often impractical or requires expert knowledge.
method Study of interactive learning with hindsight instruction feedback, introducing LORIL algorithm.
result LORIL algorithm achieves regret, improving over baselines in domains with low-rank instruction-response distribution.
HIGhER uses language to generate new instructions for better learning from mistakes.
problem Improving instruction following in reinforcement learning environments.
method Hindsight Generation for Experience Replay (HIGhER) that learns from mistakes and relabels episodes.
result HIGhER enhances instruction following in reinforcement learning environments.
The paper improves HER by prioritizing virtual goals and removing misleading samples.
problem Sparse reward functions in reinforcement learning.
method Prioritizing virtual goals based on instructiveness and removing misleading samples.
result Significant improvement in success rate and sample efficiency.
HGAIL learns policies without expert demonstrations.
problem Lack of expert demonstrations in imitation learning.
method Combines hindsight and GAIL to learn policies.
result Comparable performance to current methods, with curriculum learning.
New algorithms learn POMDPs efficiently with hindsight observability.
problem Hardness of learning in POMDPs due to partial observability.
method Hindsight Observable Markov Decision Process (HOMDP) and new algorithms for tabular and function approximation settings.
result Sample-efficient learning in POMDPs with optimal dependence on latent state and observation cardinalities.
CDP improves RL performance and sample-efficiency by prioritizing rare goal states.
problem Learning from imbalanced data in RL.
method Curiosity-Driven Prioritization (CDP) framework.
result CDP improves both performance and sample-efficiency of RL agents.
We optimize rebalancing options by limiting asset allocations to a few choices, reducing the price and guaranteeing near-optimal performance.
problem Optimizing rebalancing strategies under discrete hindsight optimization.
method Restricting the set of rebalancing rules to a small number of asset allocations.
result Guaranteed near-optimal performance with a rock-bottom option price.
New algorithms assign credit to past decisions based on hindsight.
problem Efficient credit assignment in reinforcement learning.
method Explicit credit assignment using new data in hindsight.
result Value functions can be rewritten to include hindsight credit assignment.
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
Generalized Hindsight improves RL by reusing data from one task for another.
problem High sample complexity in reinforcement learning due to wasted uninformative data.
method Approximate inverse reinforcement learning to relabel behaviors with better-suited tasks.
result Efficient reuse of samples, reducing sample complexity on multi-task RL tasks.
Efficient RL in partially observable risk-sensitive environments with hindsight observations.
problem Risk-sensitive reinforcement learning in partially observable environments.
method Integrates hindsight observations into POMDP framework, develops novel RL algorithm.
result Achieves polynomial regret with provable efficiency, outperforming existing methods.
Paper introduces a new regret measure for online convex optimization with smooth losses.
problem Online convex optimization with smooth losses.
method Introduces a new regret measure that depends on the cumulative squared gradient norm.
result The regret can be arbitrarily sharper than existing measures when losses have vanishing curvature.
Improved Q-learning with hindsight factor reduces overestimation errors.
problem Overestimation errors in Q-learning leading to suboptimal policies.
method Introducing a hindsight factor as an additional loss term in Q-learning.
result The hindsight factor significantly reduces overestimation errors and improves stability.
In E-commerce advertising, where product recommendations and product ads are presented to users simultaneously, the traditional setting is to display ads at fixed positions. However, under such a setting, the advertising system loses the flexibility to control the number and positions of ads, resulting in sub-optimal p…
Dynamical-VAE learns causal dynamics from POMDPs using future information.
problem Learning accurate state representations from partial observations in POMDPs.
method Dynamical Variational Auto-Encoder (DVAE) with hindsight framework.
result DVAE uncovers causal graph more effectively than history-based methods.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
This paper derives a robust on-line equity trading algorithm that achieves the greatest possible percentage of the final wealth of the best pairs rebalancing rule in hindsight. A pairs rebalancing rule chooses some pair of stocks in the market and then perpetually executes rebalancing trades so as to maintain a target …
HL algorithms improve resource allocation in cloud environments.
problem Sequential decision-making under uncertainty with exogenous variables.
method HL algorithms leverage exogenous variable samples to infer counterfactual consequences.
result HL algorithms outperform classic methods and reinforcement learning in resource allocation.
GDT improves reinforcement learning by matching future state information efficiently.
problem Efficient learning of multi-task policies from trajectory data.
method Generalized Decision Transformer (GDT) for offline hindsight information matching.
result GDT enables effective offline multi-task state-marginal matching and imitation learning.
A new EM framework for goal-conditioned RL improves performance on sparse reward tasks.
problem Handling sparse rewards in goal-conditioned reinforcement learning.
method A graphical model framework with an EM algorithm that includes a learning-in-hindsight E-step and a supervised M-step.
result hEM significantly outperforms model-free baselines on goal-conditioned benchmarks with sparse rewards.
Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functio…
This paper prices and replicates the financial derivative whose payoff at is the wealth that would have accrued to a $\$1$ deposit into the best continuously-rebalanced portfolio (or fixed-fraction betting scheme) determined in hindsight. For the single-stock Black-Scholes market, Ordentlich and Cover (1998) only p…
Improves sparse reinforcement learning efficiency with OYMB.
problem Sparse rewards hinder reinforcement learning performance.
method Introduces OYMB, a sampler for HER to control minibatch makeup.
result HER combined with OYMB leads to faster real goal completion.
This work improves RL for complex robotic tasks by guiding exploration with task-specific goal distributions.
problem Solving long-horizon, complex sequential tasks in robotics with sparse rewards.
method Extends hindsight relabelling to task-specific goal distributions using a small set of demonstrations.
result Significantly higher overall performance on complex robotic manipulation tasks.
Improves reinforcement learning for complex tasks with sparse feedback.
problem Learning optimal policies from sparse feedback is challenging.
method Three algorithms based on Hindsight Experience Replay (HER) to improve performances.
result Vast improvement in final success rate and sample efficiency.
This note improves on universal portfolios by using bilinear strategies.
problem Improving on the best constant-rebalanced portfolio.
method Generalizing the best bilinear trading strategy using performance-weighted averaging.
result A universal bilinear portfolio that asymptotically dominates the original universal portfolio.
New method improves continual learning by anchoring past knowledge.
problem Catastrophic forgetting in continual learning.
method Bilevel optimization to update current task knowledge while keeping past task predictions.
result Improves accuracy and forgetting metrics compared to experience replay.
Develops a new approach to learn value predictors from data.
problem Learning effective value predictors from data in reinforcement learning.
method Combines model learning and model-free methods to learn what to model.
result Significantly improves value prediction in simple and complex domains.
MO2 learns useful behaviours from past experience for new tasks.
problem Discovering useful behaviours from past experience and transferring them to new tasks.
method Model-Based Offline Options (MO2) framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces.
result MO2 outperforms recent option learning methods on complex long-horizon continuous control tasks.
HO2 learns options from data efficiently, improving robot manipulation tasks.
problem Learning options from raw pixel inputs in 3D robot manipulation tasks.
method HO2 infers likely option choices and trains all policy components off-policy.
result HO2 outperforms existing methods on 3D robot manipulation tasks.
The paper explores how planning with models improves credit assignment in reinforcement learning.
problem Improving credit assignment in reinforcement learning.
method Investigates the use of forward and backward models for planning in reinforcement learning.
result Establishes the relative merits and limitations of forward and backward planning mechanisms.
Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of…
Adaptive designs achieve strong Neyman regret guarantees for ATE estimation.
problem Estimating unbiased average treatment effect in sequential experiments.
method Proposed adaptive designs with Neyman regret under boundedness assumptions and multigroup Neyman regret in covariate-based settings.
result Adaptive designs outperform non-adaptive designs in terms of Neyman regret, especially in covariate-based settings.
Motivated by the pricing of lookback options in exponential Lévy models, we study the difference between the continuous and discrete supremum of Lévy processes. In particular, we extend the results of Broadie et al. (1999) to jump-diffusion models. We also derive bounds for general exponential Lévy models.
In many practical problems, a learning agent may want to learn the best action in hindsight without ever taking a bad action, which is significantly worse than the default production action. In general, this is impossible because the agent has to explore unknown actions, some of which can be bad, to learn better action…
In this paper, we consider an online optimization process, where the objective functions are not convex (nor concave) but instead belong to a broad class of continuous submodular functions. We first propose a variant of the Frank-Wolfe algorithm that has access to the full gradient of the objective functions. We show t…