New approach decouples skill learning and language grounding for autonomous agents.
problem Autonomous acquisition of skills without external instructions and feedback.
method Language-Goal-Behavior (LGB) architecture with semantic representation.
result Decouples skill learning and language grounding, enabling diversity and strategy switching.
Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward function…
This paper proposes a new method to connect language and physical actions in reinforcement learning.
problem Connecting linguistic representations to the physical world in embodied agents.
method Language-conditioned goal generators to decouple sensorimotor learning from language acquisition.
result Agents can demonstrate a diversity of behaviors for any given instruction.
NetHack Learning Environment (NLE) tests RL algorithms, offering scalable, complex, and challenging gameplay.
problem Challenging environments for testing RL algorithms.
method Procedurally generated, stochastic, rich, and complex NetHack environment.
result Demonstrates empirical success for early stages of NetHack using RL.
Paper introduces TrufLL for language model training without labeled data.
problem Training language models from scratch with reinforcement learning.
method Dynamic vocabulary truncation in RL for language generation.
result TrufLL successfully learns language generation policies from scratch.
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language underst…
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
Reincarnating RL reuses prior work to accelerate RL progress.
problem Efficiency and accessibility in reinforcement learning for large-scale applications.
method Transfer of learned policies between RL agents or design iterations, focusing on value-based RL.
result Demonstrated gains in performance over tabula rasa RL on various tasks.
Paper shows RLHF can be solved similarly to standard RL.
problem Difficulty of RLHF compared to standard RL.
method Reduction to reward-based RL techniques.
result RLHF can be solved using existing algorithms for reward-based RL.
RL tackles decision making in unknown environments, focusing on efficiency and efficacy.
problem Efficiency and efficacy in RL algorithms for sample-starved situations.
method Markov Decision Processes, model-based and value-based approaches, policy optimization.
result Enhanced understanding and improvements in sample and computational efficacies of RL algorithms.
Catalyst.RL accelerates RL research with efficient training.
problem Efficient reinforcement learning training in complex environments.
method Open-source PyTorch framework with distributed training and RL algorithms.
result Catalyst.RL achieved 2nd place in a computationally expensive RL challenge.
Study shows effectiveness of offline RL in online RL tasks.
problem Improving online RL efficiency using offline RL data.
method Formalized framework for incorporating offline RL as online RL subroutines, introducing techniques to enhance effectiveness.
result Effectiveness of the framework depends on task nature, techniques greatly enhance effectiveness, and existing methods are ineffective.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
RL research overhypes potential but lacks deployable solutions.
problem Current RL research overhypes potential and lacks deployable solutions.
method Identifies and critiques current RL research practices.
result Current RL research direction is unlikely to lead to practical, economically viable solutions.
RL applied to finance tasks, highlighting challenges and future directions.
problem Decision-making tasks in finance using RL.
method Meta-analysis of RL applications, identifying challenges and proposing future directions.
result Challenges in RL performance and future research directions.
Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn. Given the importance of learning in our intelligence, RL has been thought to be one of key compone…
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
MiniHack simplifies creation of complex RL environments.
problem Limited availability of challenging RL benchmarks.
method Develops a sandbox framework for easy RL environment design.
result MiniHack enables rapid creation of diverse RL testbeds.
Generative flow networks use RL to learn probabilistic models efficiently.
problem Training generative models with RL for compositional discrete objects.
method Reformulate GFlowNet training as entropy-regularized RL with specific reward and regularizer.
result Entropy-regularized RL can be competitive with established GFlowNet training methods.
Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning…
RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
Despite the recent progress in deep reinforcement learning field (RL), and, arguably because of it, a large body of work remains to be done in reproducing and carefully comparing different RL algorithms. We present catalyst.RL, an open source framework for RL research with a focus on reproducibility and flexibility. Ma…
Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the re…
HTMRL uses HTM for RL, adapting faster to changing environments.
problem Adapting to non-stationary environments in RL.
method Strictly HTM-based RL algorithm.
result HTMRL adapts faster to changing environments in a 10-armed bandit.
We discuss deep reinforcement learning in an overview style. We draw a big picture, filled with details. We discuss six core elements, six important mechanisms, and twelve applications, focusing on contemporary work, and in historical contexts. We start with background of artificial intelligence, machine learning, deep…
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
M3PO improves model-based meta-RL with theoretical guarantees.
problem Improving sample efficiency in multi-task RL with theoretical guarantees.
method Extending Janner et al. (2019) theorems, proposing M3PO with performance guarantees.
result M3PO outperforms existing methods in continuous-control benchmarks.
evo-RL combines evolutionary computation with reinforcement learning for better adaptability.
problem Improving reinforcement learning algorithms' adaptability and performance in environments with rewardless states.
method Embedding reinforcement learning in an evolutionary cycle, distinguishing instinctive from learnable behavior.
result evo-RL leads to state-of-the-art performance on OpenAI Gym control problems with rewardless states.
This work bridges offline RL and DRL to address distributional shift.
problem Distributional shift in offline RL due to difference in state-action visitation distributions.
method Proposes offline RL algorithms using DRL framework, characterizes sample complexity under single policy concentrability.
result Demonstrates superior performance of proposed algorithms through simulations.
Effective network congestion control strategies are key to keeping the Internet (or any large computer network) operational. Network congestion control has been dominated by hand-crafted heuristics for decades. Recently, ReinforcementLearning (RL) has emerged as an alternative to automatically optimize such control str…
RL Unplugged benchmarks offline RL methods across diverse domains.
problem Evaluate offline reinforcement learning methods without online data collection.
method Proposes a benchmark suite with diverse datasets and detailed evaluation protocols.
result Demonstrates the effectiveness of offline RL methods across various domains.
Optimistic RL algorithms are simplified for deep RL with competitive performance.
problem Achieving accurate optimism in model-based RL for large-scale problems.
method Interpreting scalable optimistic model-based algorithms as solving a tractable noise augmented MDP.
result Competitive regret bound of i l d e O ( ∣ S ∣ H ∣ A ∣ T ) ilde{\mathcal{O}}( |\mathcal{S}|H\sqrt{|\mathcal{A}| T } ) i l d e O ( ∣ S ∣ H ∣ A ∣ T ) for Gaussian noise augmentation. Paper proposes a hybrid RL algorithm that combines offline and online data without needing reward info.
problem How to efficiently use online data to improve RL policies using only offline data.
method A three-stage hybrid RL algorithm that uses reward-agnostic exploration and model-based offline RL.
result The hybrid RL algorithm outperforms both pure offline and pure online RL in sample complexity.
Critic-regularized regression improves offline RL performance.
problem Poor performance of off-policy algorithms in offline RL.
method Critic-regularized regression (CRR) for policy learning from fixed datasets.
result CRR outperforms state-of-the-art offline RL algorithms significantly.
This paper surveys RL for combinatorial optimization, focusing on TSP.
problem Optimizing solutions for combinatorial optimization problems.
method Reinforcement learning applied to combinatorial optimization problems, specifically the TSP.
result Deep learning mechanisms enhance RL algorithms for near-optimal solutions.
Mixed RL improves RL efficiency with dual representations.
problem Poor sampling efficiency in RL methods.
method Uses dual representations of environmental dynamics to search optimal policies.
result Proves convergence and recursive stability of the mixed RL.
The standard reinforcement learning (RL) formulation considers the expectation of the (discounted) cumulative reward. This is limiting in applications where we are concerned with not only the expected performance, but also the distribution of the performance. In this paper, we introduce micro-objective reinforcement le…
Survey of RL algorithms for dynamic environments.
problem Handling changing operating conditions in RL.
method Survey of algorithms developed for non-stationary environments.
result Methods to adapt to varying operating conditions.
This chapter reviews statistical tools for reinforcement learning.
problem Applying RL algorithms in healthcare and ride-sharing platforms.
method Statistical inference tools for RL, including hypothesis testing and confidence interval construction.
result Highlighting the value of statistical inference in RL for both communities.
Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
HyperAgent improves RL exploration in large-scale problems.
problem Efficient exploration in large-scale reinforcement learning problems.
method Hypermodel framework for incremental posterior approximation without conjugacy.
result HyperAgent achieves logarithmic per-step computational complexity and sublinear regret.
Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.
problem Learning sparse Boolean functions with Transformers.
method Reinforcement Learning (RL) with process rewards and Supervised Fine-Tuning (SFT).
result RL learns the whole CoT chain simultaneously, while SFT learns step by step.
Rewriting history improves RL algorithms for solving multiple tasks.
problem Improving sample efficiency in multi-task reinforcement learning.
method Introducing hindsight relabeling as inverse RL to generalize goal-relabeling techniques.
result Relabeling data using inverse RL accelerates learning in multi-task settings.
A3RL combines online and offline RL with active sampling to improve policy learning.
problem Combining online and offline RL for sample efficiency and robustness.
method A3RL uses a confidence-aware Active Advantage Aligned (A3) sampling strategy to prioritize data from both online and offline sources.
result A3RL outperforms competing online RL techniques that use offline data.
RH-UCRL combines pessimism and optimism for robust RL.
problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.
method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.
result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.