New method improves meta-reinforcement learning efficiency.
problem Sample inefficiency in meta-reinforcement learning.
method Hindsight Foresight Relabeling (HFR) method.
result HFR improves performance on various meta-reinforcement learning tasks.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “hindsight relabelling”
Multi-task reinforcement learning (RL) aims to simultaneously learn policies for solving many tasks. Several prior works have found that relabeling past experience with different reward functions can improve sample efficiency. Relabeling methods typically ask: if, in hindsight, we assume that our experience was optimal…
One of the key reasons for the high sample complexity in reinforcement learning (RL) is the inability to transfer knowledge from one task to another. In standard multi-task RL settings, low-reward data collected while trying to solve one task provides little to no signal for solving that particular task and is hence ef…
This work improves RL for complex robotic tasks by guiding exploration with task-specific goal distributions.
problem Solving long-horizon, complex sequential tasks in robotics with sparse rewards.
method Extends hindsight relabelling to task-specific goal distributions using a small set of demonstrations.
result Significantly higher overall performance on complex robotic manipulation tasks.
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language underst…
Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failed experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and un…
Identifies minimal training subset to flip a prediction.
problem Flipping predictions in machine learning models.
method Extended influence function for relabeling minimal subset.
result Relabeling fewer than 2% of training points can flip a prediction.
Cincer cleans both new and past data by identifying and relabeling suspicious and counter-examples.
problem Sequential learning under label noise, especially in applications with human supervision.
method Cincer uses example-based explanations to identify and relabel suspicious and counter-examples, leveraging Fisher information matrix approximation.
result Cincer achieves better data and models by clarifying the model's suspicions, especially with FIM approximation.
Paper proposes methods to improve SVM classifiers in noisy data scenarios.
problem Improving SVM classifiers when training data contains label noise.
method Mixed Integer Linear and Non Linear models with relabeling and clustering.
result Effective methods improve SVM performance in noisy data scenarios.
New algorithms learn POMDPs efficiently with hindsight observability.
problem Hardness of learning in POMDPs due to partial observability.
method Hindsight Observable Markov Decision Process (HOMDP) and new algorithms for tabular and function approximation settings.
result Sample-efficient learning in POMDPs with optimal dependence on latent state and observation cardinalities.
Reinforcement Learning(RL) with sparse rewards is a major challenge. We propose \emph{Hindsight Trust Region Policy Optimization}(HTRPO), a new RL algorithm that extends the highly successful TRPO algorithm with \emph{hindsight} to tackle the challenge of sparse rewards. Hindsight refers to the algorithm's ability to l…
Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising way to train an agent to perform goal-directed tasks is to use Hindsight Learning approaches. In thes…
We study T. Cover's rebalancing option (Ordentlich and Cover 1998) under discrete hindsight optimization in continuous time. The payoff in question is equal to the final wealth that would have accrued to a $\$1$ deposit into the best of some finite set of (perhaps levered) rebalancing rules determined in hindsight. A r…
Framework uses hindsight regret to audit marketing budget allocations.
problem Lack of principled way to assess strategic budget allocations.
method Hindsight regret framework based on constraint-faithful benchmark.
result Identifies practical trade-off between allocation flexibility and detectability.
Efficient RL in partially observable risk-sensitive environments with hindsight observations.
problem Risk-sensitive reinforcement learning in partially observable environments.
method Integrates hindsight observations into POMDP framework, develops novel RL algorithm.
result Achieves polynomial regret with provable efficiency, outperforming existing methods.
Solving goal-oriented tasks is an important but challenging problem in reinforcement learning (RL). For such tasks, the rewards are often sparse, making it difficult to learn a policy effectively. To tackle this difficulty, we propose a new approach called Policy Continuation with Hindsight Inverse Dynamics (PCHID). Th…
Interactive learning with hindsight instruction feedback achieves better performance than traditional methods.
problem Learning with expert supervision of optimal responses is often impractical or requires expert knowledge.
method Study of interactive learning with hindsight instruction feedback, introducing LORIL algorithm.
result LORIL algorithm achieves regret, improving over baselines in domains with low-rank instruction-response distribution.
Improved anomaly detection for launch vehicle propulsion systems using LSTM and statistical relabeling.
problem Detecting anomalies in real-time telemetry data for launch vehicles.
method Utilized LSTM networks for anomaly classification, introduced a statistical detector based on Mahalanobis distance and forward-backward detection fractions to adjust training labels.
result Precision and recall of the LSTM classifier improved by 7% and 22% respectively after statistical relabeling.
Algorithm learns new tasks efficiently from past experience.
problem Lack of robustness to distributional shift in meta-reinforcement learning.
method Model Identification and Experience Relabeling (MIER) using dynamics models.
result Efficient extrapolation to out-of-distribution tasks.
Dynamical-VAE learns causal dynamics from POMDPs using future information.
problem Learning accurate state representations from partial observations in POMDPs.
method Dynamical Variational Auto-Encoder (DVAE) with hindsight framework.
result DVAE uncovers causal graph more effectively than history-based methods.
New method uses hindsight to make exploration robust in stochastic environments.
problem Exploration in sparse-reward or reward-free environments, especially in stochastic settings.
method Learn representations of the future that capture unpredictable aspects, using them to predict and reward only the predictable parts of the world.
result Improves exploration in Atari games and Montezuma's Revenge, robust to stochasticity.
This paper derives a robust on-line equity trading algorithm that achieves the greatest possible percentage of the final wealth of the best pairs rebalancing rule in hindsight. A pairs rebalancing rule chooses some pair of stocks in the market and then perpetually executes rebalancing trades so as to maintain a target …
Study matches two noisy point clouds with geometric transformations and relabeling.
problem Matching two noisy point clouds with orthogonal transformations and relabeling.
method Information-theoretic results and Ping-Pong algorithm for computational alignment.
result The Ping-Pong algorithm retrieves the planted signal after one step.
Deep neural networks (DNNs) are powerful tools in computer vision tasks. However, in many realistic scenarios label noise is prevalent in the training images, and overfitting to these noisy labels can significantly harm the generalization performance of DNNs. We propose a novel technique to identify data with noisy lab…
Goal-oriented reinforcement learning has recently been a practical framework for robotic manipulation tasks, in which an agent is required to reach a certain goal defined by a function on the state space. However, the sparsity of such reward definition makes traditional reinforcement learning algorithms very inefficien…
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Dealing with sparse rewards is a longstanding challenge in reinforcement learning. The recent use of hindsight methods have achieved success on a variety of sparse-reward tasks, but they fail on complex tasks such as stacking multiple blocks with a robot arm in simulation. Curiosity-driven exploration using the predict…
HL algorithms improve resource allocation in cloud environments.
problem Sequential decision-making under uncertainty with exogenous variables.
method HL algorithms leverage exogenous variable samples to infer counterfactual consequences.
result HL algorithms outperform classic methods and reinforcement learning in resource allocation.
A new EM framework for goal-conditioned RL improves performance on sparse reward tasks.
problem Handling sparse rewards in goal-conditioned reinforcement learning.
method A graphical model framework with an EM algorithm that includes a learning-in-hindsight E-step and a supervised M-step.
result hEM significantly outperforms model-free baselines on goal-conditioned benchmarks with sparse rewards.
When Daan Krammer and Stephen Bigelow independently proved that braid groups are linear, they used the Lawrence-Krammer-Bigelow representation for generic values of its variables q and t. The t variable is closely connected to the traditional Garside structure of the braid group and plays a major role in Krammer's alge…
In Hindsight Experience Replay (HER), a reinforcement learning agent is trained by treating whatever it has achieved as virtual goals. However, in previous work, the experience was replayed at random, without considering which episode might be the most valuable for learning. In this paper, we develop an energy-based fr…
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…
Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functio…
Compared to reinforcement learning, imitation learning (IL) is a powerful paradigm for training agents to learn control policies efficiently from expert demonstrations. However, in most cases, obtaining demonstration data is costly and laborious, which poses a significant challenge in some scenarios. A promising altern…
This paper prices and replicates the financial derivative whose payoff at is the wealth that would have accrued to a $\$1$ deposit into the best continuously-rebalanced portfolio (or fixed-fraction betting scheme) determined in hindsight. For the single-stock Black-Scholes market, Ordentlich and Cover (1998) only p…
PEAR dynamically reconfigures agent roles to prevent persistent biases in multi-agent debates.
problem Persistent positional biases and sensitivity to role assignments in fixed topologies.
method Dynamic reconfiguration of agent roles and sparse topologies based on evolving agent states.
result Significantly improves average accuracy over debate baselines across multiple reasoning benchmarks.
We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihood of them having led to the observed outcome. This approach uses new information in hindsight, rathe…
New algorithm combines curriculum learning with HER for complex object manipulation tasks.
problem Learning complex sequential object manipulation tasks from scratch is challenging.
method Curriculum learning with Hindsight Experience Replay (HER) for recurrent object manipulation tasks.
result Significant improvement in learning sequential object manipulation tasks compared to vanilla-HER.
When learning from positive and unlabelled data, it is a strong assumption that the positive observations are randomly sampled from the distribution of conditional on , where X stands for the feature and Y the label. Most existing algorithms are optimally designed under the assumption. However, for many real…
This note provides a neat and enjoyable expansion and application of the magnificent Ordentlich-Cover theory of "universal portfolios." I generalize Cover's benchmark of the best constant-rebalanced portfolio (or 1-linear trading strategy) in hindsight by considering the best bilinear trading strategy determined in hin…
FairUDT uses uplift decision trees to detect and mitigate discrimination in training data.
problem Bias in machine learning classifiers due to historical discrimination or underrepresentation of minority groups.
method Integrates uplift modeling with decision trees and introduces a modified leaf relabeling approach for fairness.
result Achieves an acceptable accuracy-discrimination tradeoff while maintaining interpretability.
GDT improves reinforcement learning by matching future state information efficiently.
problem Efficient learning of multi-task policies from trajectory data.
method Generalized Decision Transformer (GDT) for offline hindsight information matching.
result GDT enables effective offline multi-task state-marginal matching and imitation learning.
Paper introduces a new regret measure for online convex optimization with smooth losses.
problem Online convex optimization with smooth losses.
method Introduces a new regret measure that depends on the cumulative squared gradient norm.
result The regret can be arbitrarily sharper than existing measures when losses have vanishing curvature.
HO2 learns options from data efficiently, improving robot manipulation tasks.
problem Learning options from raw pixel inputs in 3D robot manipulation tasks.
method HO2 infers likely option choices and trains all policy components off-policy.
result HO2 outperforms existing methods on 3D robot manipulation tasks.
The paper explores how planning with models improves credit assignment in reinforcement learning.
problem Improving credit assignment in reinforcement learning.
method Investigates the use of forward and backward models for planning in reinforcement learning.
result Establishes the relative merits and limitations of forward and backward planning mechanisms.
Motivated by the pricing of lookback options in exponential Lévy models, we study the difference between the continuous and discrete supremum of Lévy processes. In particular, we extend the results of Broadie et al. (1999) to jump-diffusion models. We also derive bounds for general exponential Lévy models.
The study examines how permutation-based optimization performance varies across different function representations.
problem Understanding how the order of function evaluations affects optimization performance.
method Iterative search setting with sampling without replacement, algebraic function recombination, correlation analysis, hierarchical clustering, PCA, ANOVA.
result Algebraically modified benchmarks yield stable re-rankings and coherent clusters of functions and sampling policies, indicating non-additive search effort.
Hindsight Experience Replay (HER) is a multi-goal reinforcement learning algorithm for sparse reward functions. The algorithm treats every failure as a success for an alternative (virtual) goal that has been achieved in the episode. Virtual goals are randomly selected, irrespective of which are most instructive for the…