Rewriting history improves RL algorithms for solving multiple tasks.
problem Improving sample efficiency in multi-task reinforcement learning.
method Introducing hindsight relabeling as inverse RL to generalize goal-relabeling techniques.
result Relabeling data using inverse RL accelerates learning in multi-task settings.
Paper uses RL and diffusion models to solve Bayesian inverse problems.
problem Bayesian inverse problems with latent biases.
method Relative Trajectory Balance (RTB) for RL, conditional diffusion models, off-policy backtracking exploration.
result RTB improves diffusion model posteriors for inverse problems.
Adversarial RL recovers agent rewards from financial market data simulations.
problem Recovering agent rewards in volatile financial markets with unknown dynamics.
method Adversarial inverse reinforcement learning in latent space simulations.
result Adversarial RL can robustly recover agent rewards from latent space representations of real market data.
This work simplifies IRL by using potential-based reward shaping.
problem Computational inefficiency in inverse reinforcement learning.
method Potential-based reward shaping to reduce RL sub-problems.
result Reduces computational burden of inverse reinforcement learning.
Paper models human motivations in games using inverse RL.
problem Lack of understanding human motivations in game models.
method Multi-Motivation Behavior Modeling (MMBM) using inverse reinforcement learning.
result Reveals significant differences in value structures among player groups.
Robo-advisor uses ML to optimize investment performance.
problem Maximizing investment performance with historical data.
method Inverse optimization and deep reinforcement learning.
result Robo-advisor consistently outperformed S&P 500.
Combines human and AI to optimize fund managers' investment decisions.
problem Improving fund managers' investment practices.
method Combines Inverse Reinforcement Learning and Reinforcement Learning.
result Improves fund managers' investment performance.
RECODE uses clustering and embedding to track state visitation counts in RL.
problem Efficient novelty-based exploration in nonstationary RL environments.
method Non-parametric clustering, online density estimation, inverse dynamics loss.
result RECODE achieves state-of-the-art performance in challenging RL tasks.
The QLBS model is a discrete-time option hedging and pricing model that is based on Dynamic Programming (DP) and Reinforcement Learning (RL). It combines the famous Q-Learning method for RL with the Black-Scholes (-Merton) model's idea of reducing the problem of option pricing and hedging to the problem of optimal reba…
Generalized Hindsight improves RL by reusing data from one task for another.
problem High sample complexity in reinforcement learning due to wasted uninformative data.
method Approximate inverse reinforcement learning to relabel behaviors with better-suited tasks.
result Efficient reuse of samples, reducing sample complexity on multi-task RL tasks.
Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure. In this work, we present a decoupled learning strategy for RL that creates a shared representation space where knowledge can be robustly transferred. …
Improves imitation learning in RL by learning reward function efficiently.
problem Lack of effective reward function approximation in AIRL for imitation tasks.
method Proposes Off-Policy AIRL that combines adversarial learning with efficient reward function approximation.
result Shows superior imitation performance and efficiency compared to state-of-the-art AIL algorithms.
This paper tackles adversarial reinforcement learning in cyber defence with partial observability.
problem Adversarial manipulation of reinforcement learning agents in autonomous cyber defence.
method Proposes an inversion defence method to counteract causative attacks under partial observability.
result The proposed inversion defence method effectively reduces the impact of adversarial attacks without affecting non-attack training scenarios.
Study shows how to motivate AI agents like humans through learning from demonstrations.
problem Transfer human-like intrinsic motivation to AI agents in RL.
method Inverse RL approach to learn an exploration bonus from demonstrations.
result AI agents can learn and use complex exploration behaviors from demonstrations.
Framework uses IRL and RL to elicit and optimize risk preferences robustly to noise.
problem Eliciting and optimizing risk preferences in noisy environments.
method Adaptive Bayesian IRL for elicitation, model-free RL for optimization, using quantile networks.
result Framework achieves convergence rate of O ( exp ( − c m + O ( m log m ) ) ) O(\exp(-cm+O(\sqrt{m\log m}))) O ( exp ( − c m + O ( m log m ))) . EQO uses a simple bonus term for efficient exploration in tabular RL.
problem Achieving minimax optimal reinforcement learning with practical efficiency.
method Exploration via Quasi-Optimism, employing a bonus proportional to state-action visit count.
result Achieves the sharpest known regret bound for tabular RL.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.
A paper uses RL to design microfluidic flow shapes efficiently.
problem Designing complex flow shapes in microfluidics using inverse problems.
method Formulated as a Reinforcement Learning (RL) problem, trained a DoubleDQN agent.
result Success frequency reached 90% in 200,000 episodes, rewards converged.
Paper tackles efficient IRL in offline settings with polynomial samples and runtime.
problem Efficiently learning reward functions from expert demonstrations in offline settings.
method Adapting the pessimism principle for offline RL, achieving strong guarantees.
result Achieved efficient IRL in offline and online settings with polynomial samples and runtime.
PCHID improves sample efficiency in reinforcement learning tasks.
problem Sparse rewards make learning policies difficult in reinforcement learning.
method Hindsight Inverse Dynamics with Hindsight Experience Replay and Policy Continuation.
result PCHID significantly improves sample efficiency and final performance on multi-goal tasks.
Paper improves RLS for sparse outlier detection in linear models.
problem Outliers contaminate linear regression models infrequently.
method Hierarchical-optimization recursive least squares with sparsity-inducing regularization.
result The method robustly estimates linear filters/systems with outliers.
Study improves feature acquisition for static settings in AFAPE.
problem Evaluate AFAPE performance in static feature settings.
method Derive and adapt IPW, DM, and DRL estimators for MAR and MNAR missingness.
result Improved data efficiency in synthetic and real-world experiments.
New algorithm reduces performance loss in IRL with mismatched transition dynamics.
problem Performance degradation in inverse reinforcement learning due to mismatched transition dynamics.
method Proposed a robust Maximum Causal Entropy (MCE) IRL algorithm leveraging robust reinforcement learning insights.
result Empirically demonstrated stable performance improvement under transition dynamics mismatches.
Paper presents a rank-1 approximation method for natural policy gradients in deep RL.
problem Computing natural gradients requires inverting the Fisher Information Matrix, which is computationally expensive.
method Develops a rank-1 approximation to the inverse Fisher Information Matrix for efficient natural policy optimization.
result The rank-1 approximation converges faster and has similar sample complexity to stochastic policy gradient methods.
RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.
problem Large linear systems in portfolio optimization and option pricing lead to slow convergence.
method Reinforcement Learning (RL) dynamically adjusts block-preconditioner sizes to accelerate convergence.
result RL-driven solver significantly reduces computational cost and accelerates convergence.
Method distills reward and strategies from diverse demonstrators.
problem Reward ambiguity and heterogeneity in human demonstrations.
method Reward network distillation to infer task goal and strategies.
result Better recovery of task and strategy rewards.
RIDM combines imitation and RL with a single demo, no action info needed.
problem Learning from a single observed demonstration without action information.
method Reinforced Inverse Dynamics Modeling (RIDM) that operates on raw state features.
result RIDM performs favorably compared to baseline on simulated and real tasks.
Paper tackles sample-efficient RL for linearly realizable MDPs with limited revisiting.
problem Sample-efficient reinforcement learning for linearly realizable MDPs with limited revisiting.
method Develops a new sampling protocol that allows for backtracking and revisiting states in a controlled manner.
result Achieves polynomial sample complexity scaling with feature dimension, horizon, and inverse sub-optimality gap.
No real-world reward function is perfect. Sensory errors and software bugs may result in RL agents observing higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formal…
ABS dynamically adjusts batch size based on policy stability, improving RL performance.
problem Diminishing returns with large batch sizes in RL due to non-stationary data.
method Adaptive Batch Scaling (ABS) with Behavioral Divergence metric.
result Larger batch sizes can improve RL performance, contrary to conventional wisdom.
State-only imitation learning improves dexterous manipulation learning from videos.
problem High sample complexity in complex domains like dexterous manipulation.
method Train an inverse dynamics model to predict actions from states and train the policy jointly.
result Performs on par with state-action approaches and outperforms RL alone.
RL-LOW algorithm achieves exponential simple regret in offline RLHF with pairwise comparisons.
problem Offline reinforcement learning from human feedback with pairwise comparisons.
method Proposes RL-LOW algorithm to minimize simple regret with exponential convergence.
result Achieves an exponential form of simple regret of \(\exp ( - Ω(n/H) )\).
New RL approach handles non-exponential discounting for sequential decisions.
problem Modeling human discounting in sequential decision-making tasks.
method Generalized model-based reinforcement learning with arbitrary discount functions, using Hamilton-Jacobi-Bellman equation and collocation method.
result Validated approach on simulated problems, showing applicability to human discounting.
This survey categorizes IRL challenges and methods.
problem Inferring reward functions from observed behavior.
method Categorizes and discusses existing IRL methods.
result Challenges and solutions in IRL.
Neural-Kernel CME tackles scalability and expressiveness challenges in conditional distribution representation.
problem Scalability and expressiveness challenges in kernel conditional mean embeddings.
method Combines deep learning with CMEs using a neural network optimization framework.
result Achieves competitive and often superior performance in conditional density estimation and RL.
A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.
problem Improving offline reinforcement learning performance.
method Integrates count-based conservatism into model-based offline reinforcement learning.
result The learned policy is near-optimal and outperforms existing methods.
New method converts natural language commands into reward functions for robots.
problem Creating effective reward functions for autonomous machines.
method Language-conditioned reward learning (LC-RL) using inverse reinforcement learning.
result Model learns transferable rewards from natural language commands.
Toolbox provides datasets and metrics for state representation learning.
problem Lack of standard evaluation datasets, metrics and tasks for state representation learning.
method Provides a set of environments, data generators, robotic control tasks, metrics and tools.
result Facilitates iterative state representation learning and evaluation in reinforcement learning settings.
A new RL framework learns flexible point process models from event data.
problem Learning flexible models for complex, ordered event data in continuous time.
method Inspired by reinforcement learning, a policy is parameterized as a recurrent neural network to mimic the observed event distribution.
result The proposed RL framework efficiently learns flexible point process models and performs well in both synthetic and real data.
New method learns multiple reward functions for complex tasks.
problem Learning reward functions for tasks with multiple ways of solving.
method Combines maximum entropy approach with Dirichlet process clustering.
result Method accurately learns reward functions for complex tasks.
SQIL uses a simple reward strategy to encourage long-horizon imitation of expert demonstrations.
problem Challenges in imitation learning with high-dimensional, continuous observations and unknown dynamics.
method Imitates expert demonstrations by providing a constant reward of +1 for matching actions in demonstrated states, and 0 for all others.
result Empirically outperforms behavioral cloning and achieves competitive results compared to GAIL.
VDB constrains discriminator to improve imitation learning and GANs.
problem Stability and performance issues in adversarial learning methods.
method Information bottleneck technique to modulate discriminator's accuracy.
result Significant improvements across imitation learning, inverse RL, and GANs.
SymCircuit learns PC structure via entropy-regularized RL, improving inference efficiency and accuracy.
problem Greedy algorithms in PC structure learning lead to suboptimal solutions.
method Entropy-regularized reinforcement learning to train a learned generative policy for PC structure inference.
result SymCircuit learns the optimal policy as a tempered Bayesian posterior, improving inference efficiency and accuracy.
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
Softmax Bellman operator improves Q-function performance in RL despite sub-optimality.
problem Softmax Bellman operator's impact on value functions in RL is problematic.
method Revisited theoretical properties of softmax Bellman operator, proving convergence and overestimation reduction.
result Softmax Bellman operator leads to superior policies in practice, even outperforming double Q-learning.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
Inverse reinforcement learning (IRL) aims to explain observed strategic behavior by fitting reinforcement learning models to behavioral data. However, traditional IRL methods are only applicable when the observations are in the form of state-action paths. This assumption may not hold in many real-world modeling setting…