This paper studies poisoning attacks in episodic RL and discovers their effectiveness depends on reward bounds.
problem Understanding security threats to RL algorithms through poisoning attacks.
method Examined two types of poisoning attacks: reward and action manipulation, in bounded and unbounded reward settings.
result The effectiveness of poisoning attacks depends on reward bounds, with different attack costs and success rates.
A new reward shaping method balances learning efficiency and effectiveness for robot manipulation.
problem Efficient and effective learning in robot manipulations with system uncertainty.
method Dense2Sparse reward shaping method combining dense and sparse rewards.
result Dense2Sparse method achieves higher expected reward and better system uncertainty tolerance.
A novel RL approach learns robotic manipulation without human demonstrations.
problem Learning robotic manipulation policies efficiently and effectively.
method Introducing simulated locomotion demonstration rewards (SLDRs) to enable RL learning.
result The approach achieves higher success rates and faster learning compared to alternatives.
New method learns various data manipulation schemes for model training.
problem Improving model training with data manipulation.
method Adapts RL reward learning algorithm for data manipulation learning.
result Significant improvement in classification performance.
Proposes a method to boost deep reinforcement learning with sparse rewards.
problem Challenges in learning complex behaviors with long horizons and sparse rewards.
method Predictive coding for reward shaping.
result Achieves better learning by providing reward signals that understand environment dynamics and emphasize useful features.
Adversaries can manipulate bandit algorithms to control chosen actions.
problem Manipulating stochastic bandit algorithms to influence chosen actions.
method Proposes an attack against ε ε ε -greedy and UCB algorithms without knowing mean rewards. result Attackers can control actions with logarithmic effort, making it easy to hijack behavior.
A new method for robot manipulation tasks using imagined object goals.
problem Learning robot manipulation in sparse reward environments.
method Train objects to reach target positions, then use predictions to create a curriculum of tasks.
result Higher success rates in challenging learning scenarios compared to alternatives.
MaMiC proposes a dual curriculum for robot manipulation tasks with sparse rewards.
problem Overcoming exploratory constraints in robot manipulation tasks with sparse rewards.
method Includes a macro curriculum scheme and a micro curriculum scheme to guide learning.
result Combining macro and micro curriculum strategies improves performance in robot manipulation tasks.
New approach uses unlabeled prior data to accelerate exploration in sparse reward tasks.
problem Sparse reward tasks in reinforcement learning.
method Learn reward model from online experience, label prior data, and use concurrently.
result Rapid exploration in challenging sparse-reward domains.
This work proposes a RL approach to learn versatile robotic manipulation tasks.
problem Challenging manipulation tasks in robotics and vision.
method Reinforcement learning (RL) to combine primitive skills, no intermediate rewards, few demonstrations, and efficient skill learning.
result Versatile robotic manipulation in challenging settings with temporary occlusions and dynamic scene changes.
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…
New attack manipulates UCB algorithm, new defense algorithm reduces pseudo-regret.
problem Adversarial attacks on stochastic bandit algorithms.
method Introducing action-manipulation attacks and proposing a robust defense algorithm.
result Proposed defense algorithm reduces pseudo-regret to O(max{log T, A}).
We develop an option pricing model based on a tug-of-war game. This two-player zero-sum stochastic differential game is formulated in the context of a multi-dimensional financial market. The issuer and the holder try to manipulate asset price processes in order to minimize and maximize the expected discounted reward. W…
PixL2R maps natural language to pixel-based rewards for RL, improving sample efficiency.
problem Sparse reward settings in RL limit applicability to complex problems.
method Directly maps natural language descriptions to pixel-based rewards for guiding RL.
result Language-based rewards significantly improve sample efficiency in policy learning.
We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempt…
Classic bandit algorithms are robust to strategic manipulation as long as the total budget is small compared to the time horizon.
problem Behavior of stochastic bandit algorithms under strategic manipulation by self-interested arms.
method Analysis of three popular bandit algorithms: UCB, ε-Greedy, and Thompson Sampling.
result Regret upper bound of O(max{B, KlnT}) for all three algorithms under arbitrary adaptive manipulation.
Malicious agents can manipulate linear contextual bandits to pull desired arms with logarithmic overhead.
problem Malicious attacks on linear contextual bandit algorithms in various domains.
method Study and propose an efficient algorithm to perform adversarial attacks on linear contextual bandits.
result Malicious agents can force a linear contextual bandit algorithm to pull any desired arm T − o ( T ) T - o(T) T − o ( T ) times over a horizon of T T T steps with logarithmic modifications. TRAIL improves robot imitation learning by focusing on task-relevant features.
problem Discriminator networks learn spurious associations, providing poor reward signals.
method Constrained discriminator optimization to learn task-relevant rewards.
result TRAIL outperforms GAIL and behaviour cloning in robotic manipulation tasks.
ORIL learns a reward function from unlabeled data to improve robot learning.
problem Leveraging unlabeled data for robot learning.
method ORIL learns a reward function from demonstrator and unlabeled trajectories, annotates data, and trains an agent via offline reinforcement learning.
result ORIL consistently outperforms BC agents on various robotic tasks.
Transforming sparse outcomes into dense process rewards for efficient reinforcement learning.
problem Training RL policies to maximize sparse outcomes.
method Incentivizing policy matching state-action visitations of successful episodes.
result Significantly faster RL finetuning performance.
Recent studies have shown that reinforcement learning (RL) models are vulnerable in various noisy scenarios. For instance, the observed reward channel is often subject to noise in practice (e.g., when rewards are collected through sensors), and is therefore not credible. In addition, for applications such as robotics, …
HGG generates goals to improve sample efficiency in robotic tasks.
problem Efficiency in reinforcement learning with sparse reward signals.
method Generates valuable hindsight goals for reinforcement learning.
result Significantly improved sample efficiency over HER.
A method using competitive experience replay enhances learning from sparse rewards.
problem Learning from sparse rewards in reinforcement learning.
method Competitive experience replay method that augments sparse rewards through an exploration competition between agents.
result The method leads to faster convergence and improved task performance.
This paper tackles fair online decision-making in contextual bandits, achieving optimal performance and fairness.
problem Fairness in online decision-making systems under strategic manipulation.
method Develops algorithms for linear and smooth reward functions, maintaining fairness and optimal regret.
result Achieves nearly minimax-optimal regret with strong fairness guarantees, even in the presence of attacks.
Optimizes reward learning design for complex tasks using nonparametric methods.
problem Challenges in specifying reward functions for complex tasks.
method Models rewards and policies as nonparametric functions in RKHSs, derives risk bounds, and optimizes query design.
result Derives non-asymptotic excess risk bounds and finite sample statistical rates for reward learning.
New RL approach infers optimal policies via variational inference.
problem Manual design of reward functions for reinforcement learning.
method Variational inference for inferring policies achieving desired outcomes.
result Eliminates need for hand-crafted reward functions for diverse tasks.
New method uses explicit human demonstrations to teach missing features in reward learning.
problem Reward learning methods rely on handcrafted features, limiting their ability to adapt to new or unexplained corrections.
method Introduces human input guiding the robot from states with missing features to states without, teaching the feature explicitly and integrating it into the reward function.
result Decreases sample complexity and improves generalization of the learned reward over deep IRL baseline.
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
Algorithm learns optimal coordination for strategic agents in uncertain settings.
problem Optimizing rewards for strategic agents with private types and actions.
method Combines delaying mechanism, reward angle estimation, and LinUCB algorithm.
result Near optimal regret bound of O ~ ( T ) \tilde{O}(\sqrt{T}) O ~ ( T ) for learning optimal policy. PlanGAN uses GANs to plan efficient trajectories for multi-goal tasks in sparse reward environments.
problem Learning with sparse rewards in multi-goal environments.
method PlanGAN combines GANs to generate trajectories leading to specified goals, then combines these into a planning algorithm.
result PlanGAN achieves comparable performance to model-free RL but is 4-8 times more sample efficient.
SEMI uses multisensory incongruity to self-supervise exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning with sparse or missing rewards.
method SEMI incentivizes exploration by maximizing multisensory incongruity, measured in perception and action incongruity.
result SEMI improves sample efficiency and learns skills without external rewards.
This work tackles real-world robotic reinforcement learning challenges.
problem Limited success of reinforcement learning in real-world robotics.
method Proposes a system for autonomous real-world learning without instrumentation.
result Demonstrates a complete system that learns without human intervention.
Attackers can significantly reduce team rewards in cooperative multi-agent reinforcement learning.
problem Robustness of cooperative multi-agent reinforcement learning to adversaries.
method Novel attack method involving training a policy network and using targeted adversarial examples.
result Reduces team reward from 20 to 9.4 by attacking a single agent, reducing winning rate from 98.9% to 0%.
TrojDRL shows reinforcement learning agents vulnerable to Trojan attacks.
problem Vulnerability of reinforcement learning agents to Trojan attacks.
method Data-poisoning and in-band reward modification to augment reinforcement learning policies.
result Policies can be imperceptibly modified to perform differently when a Trojan is triggered.
Self-supervised policy adapts after deployment without rewards.
problem Generalizing reinforcement learning policies across different environments.
method Uses self-supervision to train policies in new environments without reward signals.
result Significant improvements in generalization across diverse environments.
New method adds noise to rewards to improve reinforcement learning performance.
problem Brittleness in reinforcement learning due to variance differences between states and actions.
method Adaptive Symmetric Reward Noising (ASRN) by adding Gaussian noise to rewards based on their variance.
result ASRN improves reinforcement learning performance in various tasks, including autonomous driving.
Robot learns to control itself without rewards.
problem Autonomous reinforcement learning without access to rewards.
method Distributional planning networks optimizing for an embedding space.
result Learned goal metrics enable autonomous reinforcement learning.
Robot learns to juggle two balls from 56 minutes of experience.
problem Learning high-acceleration tasks in the real-world with binary rewards.
method Designs a learning system that incorporates safety and sample efficiency.
result High-speed manipulator learns to juggle for up to 33 minutes.
This work improves RL for complex robotic tasks by guiding exploration with task-specific goal distributions.
problem Solving long-horizon, complex sequential tasks in robotics with sparse rewards.
method Extends hindsight relabelling to task-specific goal distributions using a small set of demonstrations.
result Significantly higher overall performance on complex robotic manipulation tasks.
This paper optimizes attacks on stochastic bandits and proposes defenses against them.
problem Optimizing adversarial attacks on stochastic bandit algorithms.
method Designs optimal attack strategies and proposes defense algorithms.
result Optimal attack strategies and defense algorithms achieve near perfect performance.
A new reinforcement learning method uses mutual information to encourage agents to control their environment.
problem Learning from internal drives instead of external rewards.
method Formulate an intrinsic objective as mutual information between goal states and controllable states, derive a surrogate objective for efficient optimization.
result Demonstrated the efficacy of the approach in robotic tasks.
Robotic learning without reward engineering from images.
problem Manual reward engineering for reinforcement learning in robotics.
method Learning from examples and active solicitation of labels.
result Efficient learning of robotic skills from images without manual rewards.
Learning an optimal policy from a multi-modal reward function is a challenging problem in reinforcement learning (RL). Hierarchical RL (HRL) tackles this problem by learning a hierarchical policy, where multiple option policies are in charge of different strategies corresponding to modes of a reward function and a gati…
Study intrinsic motivation for synergistic tasks in reinforcement learning.
problem Sparse-reward synergistic tasks where multiple agents must work together.
method Propose incentivizing actions that affect the world in ways not achievable individually, using either true states or a dynamics model.
result Our approach yields more efficient learning than typical methods.
Model shows how advisors can manipulate naive investors.
problem How financial advisors manipulate naive investors.
method Agent-Based Model with Nash equilibria and best response functions.
result Greediness/naivety of investors emerge naturally from the model.
Paper proposes detecting video manipulation using stream descriptors.
problem Misuse of manipulated video content.
method Binary classifiers on multimedia stream descriptors.
result Scalable approach can detect high-quality manipulations.
Paper studies attacks on bandit algorithms and shows how attackers can manipulate data to hijack behavior.
problem Potential attacks on bandit algorithms can cause catastrophic loss in real-world applications.
method Proposes a framework of offline and online attacks on bandit algorithms using convex optimization and adaptive strategies.
result Attackers can force bandit algorithms to pull target arms with high probability by manipulating data.
Reinforcement learning (RL) has advanced greatly in the past few years with the employment of effective deep neural networks (DNNs) on the policy networks. With the great effectiveness came serious vulnerability issues with DNNs that small adversarial perturbations on the input can change the output of the network. Sev…