Paper tackles sample-efficient offline RL, proposing data diversity and unified algorithms.
problem Sample-efficient learning from historical data for sequential decision-making.
method Proposes data diversity and unifies three offline RL algorithm classes: VS, RO, and PS.
result Comparable sample efficiency for VS, RO, and PS algorithms under standard assumptions.
Improved agnostic boosting with better sample efficiency.
problem Agnostic boosting's poor sample efficiency compared to Empirical Risk Minimization.
method Leverages sample reuse across rounds, guarantees better generalization.
result Substantially more sample-efficient agnostic boosting algorithm.
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstab…
Automates RL with sample-efficient hyperparameter optimization.
problem Challenges in applying deep RL due to hyperparameter sensitivity and inefficiency.
method Population-based AutoRL framework for meta-optimizing RL algorithms and architectures.
result Reduces the number of environment interactions needed for meta-optimization by up to an order of magnitude.
PHYRE benchmark tests physics learning algorithms.
problem Developing efficient algorithms for physics reasoning.
method Simple classical mechanics puzzles in a 2D environment.
result Modern algorithms perform poorly on PHYRE.
Combines BC and GAIL for efficient imitation learning.
problem Efficient imitation learning without reward signals.
method Integrates Behavior Cloning and Generative Adversarial Imitation Learning.
result Combination leads to stable and sample-efficient learning.
Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly inc…
Improves sample efficiency and generalization in vision-based RL by enhancing exploration.
problem Low sample efficiency in vision-based RL using images as observations.
method Integrates state representation learning to enhance exploration and sample diversity.
result Significant improvement in sample efficiency across various environments.
New BE dimension measure reveals rich RL problems with sample-efficient algorithms.
problem Finding sample-efficient algorithms for complex RL problems.
method Introducing Bellman Eluder (BE) dimension and designing GOLF and OLIVE algorithms.
result GOLF and OLIVE algorithms learn near-optimal policies for low BE dimension problems with polynomial samples.
This paper advances sample-efficient learning for partially observable RL by introducing B-stability and new algorithms.
problem Hard sample complexity for learning near-optimal policies in partially observable RL.
method Proposes B-stability as a unified structural condition and develops new algorithms for sample-efficient learning.
result Any B-stable PSR can be learned with polynomial samples, improving over current best complexities.
Improves sample efficiency in RL by matching model-based gradients.
problem Lack of sample efficiency in model-free RL.
method Gradient matching between model-based and model-free RL components.
result Improves sample efficiency without increasing asymptotic bias.
VRMPO improves reinforcement learning efficiency with stochastic mirror descent.
problem Improving sample efficiency in reinforcement learning.
method VRMPO algorithm with a variance-reduced policy gradient estimator and stochastic mirror descent.
result VRMPO achieves an ε-approximate first-order stationary point with only O(ε^(-3)) sample trajectories.
A streamlined DRL algorithm improves sample efficiency without entropy maximization.
problem Improving sample efficiency in off-policy DRL algorithms.
method Output normalization and non-uniform sampling.
result Proposed algorithm matches SAC's performance without entropy maximization and improves sample efficiency.
MEMEC improves sample efficiency in reinforcement learning.
problem Lack of sample efficiency in reinforcement learning.
method Proposes MEMEC, a Boltzmann policy with state-dependent temperature for more principled exploration.
result MEMEC outperforms other methods on classic RL environments and Atari games.
Deep neuroevolution, that is evolutionary policy search methods based on deep neural networks, have recently emerged as a competitor to deep reinforcement learning algorithms due to their better parallelization capabilities. However, these methods still suffer from a far worse sample efficiency. In this paper we invest…
Increasing input dimensionality improves deep RL performance and sample efficiency.
problem Real-world reinforcement learning applications often lack sufficient training data.
method Proposed an online feature extractor network (OFENet) to improve deep RL performance and sample efficiency.
result RL agents learn more efficiently with high-dimensional input representations than with lower-dimensional state observations.
Improved sample-efficient learning for non-coherent digital jamming.
problem Learning optimal jamming strategies in non-coherent digital modulation schemes without prior knowledge.
method Introduced a linear bandit algorithm that accounts for action similarities and integrates context features.
result Significantly improved convergence behavior compared to prior art.
HGG generates goals to improve sample efficiency in robotic tasks.
problem Efficiency in reinforcement learning with sparse reward signals.
method Generates valuable hindsight goals for reinforcement learning.
result Significantly improved sample efficiency over HER.
New RL approach tackles non-linear MDPs without linear assumptions.
problem Sample efficiency in RL for complex, nonlinear MDPs with continuous states.
method Introduces EPW condition to relax linear structure requirements; provides sample-efficient RL algorithm.
result EPW condition allows solving MDPs without linear assumptions, including Atari games.
Improves reinforcement learning stability and efficiency.
problem Combining stability and efficiency in reinforcement learning.
method Combines on-policy stability with off-policy sample reuse.
result Demonstrates improved performance in both theory and practice.
DAC enhances exploration in reinforcement learning with entropy regularization.
problem Improving exploration efficiency in reinforcement learning.
method Sample-aware entropy regularization using replay buffer action distributions.
result DAC significantly outperforms existing algorithms in reinforcement learning tasks.
Maximize to Explore integrates RL components for efficient policy discovery.
problem Balancing exploration and exploitation in online RL with general function approximators.
method Integrates estimation, planning, and exploration into a single objective function.
result Achieves sublinear regret for MDPs and MGs with general function approximations.
New algorithms learn POMDPs efficiently with hindsight observability.
problem Hardness of learning in POMDPs due to partial observability.
method Hindsight Observable Markov Decision Process (HOMDP) and new algorithms for tabular and function approximation settings.
result Sample-efficient learning in POMDPs with optimal dependence on latent state and observation cardinalities.
HOMER learns latent states to explore rich environments efficiently.
problem Exploration in rich observation environments with unknown latent states.
method Interleaves representation learning and strategic exploration to identify kinematic states.
result Provably efficient exploration with polynomial sample complexity in latent states and time horizon.
Improves RLHF sample efficiency by scaling reward complexity polynomially.
problem Exponential sample complexity in RLHF algorithms for skewed preferences.
method SE-POPO, an online RLHF algorithm that achieves polynomial sample complexity.
result SE-POPO outperforms existing algorithms in sample efficiency.
Faster WIND accelerates iterative BOND for LLM alignment.
problem Iterative BOND is inefficient in practice due to sample and computation inefficiency.
method Unified game-theoretic connection to self-play alignment, WIND framework with efficient algorithms.
result WIND variant achieves superior sample efficiency and faster computation.
Novel algorithm reduces sample complexity for reinforcement learning.
problem Improving sample efficiency in reinforcement learning.
method SRVR-PG algorithm with recursive variance reduction.
result Reduces sample complexity to O(1/ε3/2). New method improves sample-efficiency in neural posterior estimation using simulator gradients.
problem High-fidelity posterior estimation with complex physical simulations is time-consuming.
method Neural Posterior Estimation (NPE) with differentiable simulators and gradient information.
result Improves sample-efficiency in posterior density estimation.
New algorithm outperforms existing ones by focusing on mastering rate.
problem Low sample efficiency in learning progress-based algorithms.
method Proposes a new algorithm based on mastering rate.
result Significantly outperforms learning progress-based algorithms.
Competition aims to develop sample-efficient RL methods using human demonstrations.
problem Exponential increase in samples needed for reinforcement learning.
method Develop sample-efficient algorithms using human demonstrations.
result Introduction of MineRL ObtainDiamond task and MineRL-v0 dataset.
New framework improves sample efficiency and robustness in RL with smooth policies.
problem Sample inefficiency and lack of robustness in deep reinforcement learning.
method SR^2L framework, smoothness-inducing regularization.
result Improved sample efficiency and robustness in both on-policy and off-policy RL algorithms.
A game-theoretic approach simplifies MBRL design and improves sample efficiency.
problem Designing stable and efficient MBRL algorithms using rich function approximators.
method Develops a game-theoretic framework where MBRL is modeled as a Stackelberg game between policy and model players.
result Proposed algorithms are highly sample efficient and match asymptotic performance of model-free policy gradient.
New method reduces bias in high-dimensional action spaces for efficient reinforcement learning.
problem Large bias and difficulty in reusing old samples in high-dimensional action spaces.
method Dimension-wise IS weight clipping to control bias and adaptively manage IS weights.
result Proposed method outperforms PPO and other RL algorithms in various tasks.
MineRL Competition reduced reinforcement learning sample needs.
problem Sample inefficiency in reinforcement learning.
method Human demonstrations and imitation learning integrated into reinforcement learning algorithms.
result Top solutions used deep reinforcement learning and imitation learning.
DR-MCTS improves decision quality and sample efficiency in complex environments.
problem Improving decision quality and sample efficiency in complex environments.
method Integrates Doubly Robust off-policy estimation into Monte Carlo Tree Search (MCTS).
result DR-MCTS achieves superior performance in Tic-Tac-Toe and VirtualHome tasks.
HySRL improves RL sample efficiency with shifted-dynamics data.
problem Leveraging historical data with shifted dynamics to improve sample efficiency in RL.
method HySRL, a hybrid transfer RL algorithm that uses prior information on dynamics shift to achieve better sample complexity.
result HySRL achieves problem-dependent sample complexity and outperforms pure online RL.
Improved sample efficiency in reinforcement learning with action guidance.
problem Sample inefficiency in deep reinforcement learning with sparse, delayed, and deceptive rewards.
method Integrating a non-expert demonstrator (e.g., MCTS) into asynchronous distributed deep reinforcement learning.
result Our methods learn faster and converge to better policies on a game.
We study the computational tractability of PAC reinforcement learning with rich observations. We present new provably sample-efficient algorithms for environments with deterministic hidden state dynamics and stochastic rich observations. These methods operate in an oracle model of computation -- accessing policy and va…
New algorithm improves sample efficiency for zero-sum Markov games.
problem Improving sample efficiency for model-free algorithms in zero-sum Markov games.
method Proposes a model-free stage-based Q-learning algorithm using variance reduction techniques.
result Achieves optimal sample complexity for finding ε-optimal Nash Equilibrium.
IMPACT improves RL training speed without sacrificing sample efficiency.
problem Limited sample efficiency in scalable RL architectures.
method Proposes IMPACT, extending IMPALA with target networks, circular buffers, and truncated importance sampling.
result IMPACT achieves higher rewards and significantly reduces training time compared to IMPALA.
Improves RL efficiency with nonparametric methods.
problem High sample complexity in policy gradient algorithms.
method Nonparametric regression and density estimation for value function estimation.
result Better sample efficiency compared to existing methods.
Enhances SAC for better sample efficiency in continuous-action tasks.
problem Improving sample efficiency in soft actor-critic algorithms.
method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).
result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.
Parallelized Bayesian quadrature improves sample efficiency and inference.
problem Efficient Bayesian inference and model evidence calculation.
method Batch Bayesian quadrature with kernel recombination for parallel sampling.
result Empirically, outperforms state-of-the-art methods in various datasets.
Policy gradient aims to maximize expected return using gradient ascent.
problem Finding a policy that maximizes expected return in a given class of policies.
method Gradient ascent applied to a differentiable model of the policy, estimating the gradient of expected return.
result Policy gradient methods require on-policy data for gradient estimation, limiting sample efficiency.
HyPE improves sample efficiency in DRL by discovering objects and hierarchies of skills.
problem Poor sample efficiency in DRL methods, especially in complex tasks.
method HyPE algorithm that discovers objects and generates hypotheses about their controllability, learning a hierarchy of skills.
result HyPE learns high-scoring policies an order of magnitude faster than state-of-the-art methods.
SDPG algorithm improves sample efficiency and reward in DRL for continuous action spaces.
problem Improving sample efficiency and reward in distributional reinforcement learning for continuous action spaces.
method SDPG algorithm models return distribution using samples via reparameterization technique.
result SDPG shows better sample efficiency and higher reward in OpenAI Gym environments.
Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functio…
This paper tackles sample-efficient reinforcement learning for partially observable Markov games.
problem Learning in partially observable Markov games with incomplete information.
method A simple algorithm combining optimism and Maximum Likelihood Estimation (MLE) for self-play, and a variant of optimistic MLE for adversarial opponents.
result The proposed algorithms achieve approximate Nash, correlated, and coarse correlated equilibria in polynomial samples for weakly revealing POMGs.