Proposes a framework for sparse optimal policies in reinforcement learning.
problem Finding sparse optimal policies in reinforcement learning.
method Regularized Markov decision processes (MDPs) with specific regularization terms.
result Sufficient and necessary conditions for inducing sparse optimal policies.
Sparse PCL algorithms improve optimal policy in Tsallis entropy-regularized MDPs.
problem Sparse optimal policies in Tsallis entropy-regularized MDPs.
method Path consistency learning (PCL) algorithms for sparse entropy-regularized RL.
result Sparse PCL algorithms reduce sub-optimality compared to soft ERL, especially in high-action problems.
A new algorithm solves sparse reward tasks efficiently in robotics.
problem Sparse or misleading rewards in reinforcement learning.
method Multi-objective model-based policy optimization with three objectives.
result Multi-DEX solves sparse reward scenarios in fewer episodes than existing methods.
Self-imitation learning improves RL in sparse, episodic reward settings.
problem Suboptimal performance of RL algorithms in sparse or episodic reward settings.
method Formulate policy optimization as a divergence minimization problem using Jensen-Shannon divergence, and learn shaped rewards from experience replays.
result Our algorithm performs comparably to existing algorithms in dense reward settings and significantly better in sparse and episodic reward settings.
Exact solution for sparse-reward MDPs with minimal state space dependence.
problem Finding optimal policies for MDPs with sparse rewards and large state spaces.
method Proposes an algorithm with time complexity O(∣R∣3imes∣A∣2) and memory complexity O(∣R∣imes∣A∣) for exact computation. result Exact policy computation without state space dependency for sparse-reward MDPs.
New concept of Blackwell regret for reinforcement learning with sparse rewards.
problem Sparse rewards in long horizon MDPs.
method Formalization of myopic discount factors, value functions, and policies in terms of Blackwell optimality; introduction of Blackwell regret.
result Selecting a discount factor for zero Blackwell regret becomes arbitrarily hard in long horizon MDPs.
HTRPO tackles sparse rewards in RL with improved stability and performance.
problem Sparse rewards in reinforcement learning.
method HTRPO extends TRPO with hindsight and QKL for better policy update stability.
result HTRPO consistently outperforms TRPO and HPG in various sparse reward tasks.
RO-TD learns sparse value functions efficiently.
problem Learning sparse value functions efficiently.
method RO-TD integrates off-policy convergent gradient TD methods and online convex regularization.
result RO-TD learns sparse value functions with low computational complexity.
Sparse MDP with entropy regularization improves reinforcement learning performance.
problem Improving reinforcement learning policies with sparse and multi-modal distributions.
method Proposes a sparse Markov decision process with causal sparse Tsallis entropy regularization.
result The proposed method achieves a constant performance error bound, outperforming soft MDPs.
Bayesian RL uses deep, sparse planning for efficient exploration.
problem Efficient Bayesian reinforcement learning with sparse exploration.
method Optimism-free Bayes-adaptive algorithm with candidate policy generator.
result Lower computational complexity and higher rewards in experiments.
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
Paper tackles imitation learning with sparse rewards and heterogeneous actions.
problem Challenges of imitation learning with sparse rewards and different actions.
method Proposes a method that balances imitation and reinforcement learning objectives.
result Agent efficiently leverages sparse rewards and learns from different actions.
Sparse coding improves reinforcement learning representations.
problem Improving representation learning in reinforcement learning.
method Developed a supervised sparse coding objective for policy evaluation.
result Sparse coding representations outperform tile-coding representations.
This work improves reinforcement learning with sparse rewards by following diverse past trajectories.
problem Challenges in reinforcement learning with sparse rewards and myopic behavior.
method Proposes a trajectory-conditioned policy to learn from a memory buffer of diverse past trajectories.
result Significantly outperforms existing methods on complex tasks with local optima.
Transforming sparse outcomes into dense process rewards for efficient reinforcement learning.
problem Training RL policies to maximize sparse outcomes.
method Incentivizing policy matching state-action visitations of successful episodes.
result Significantly faster RL finetuning performance.
Proposes pT-Learning for optimal dynamic treatment regimes in mHealth.
problem Challenges in learning optimal dynamic treatment regimes with large intervention options and infinite time horizon.
method Proximal Temporal consistency Learning (pT-Learning) framework for adaptively adjusting between deterministic and stochastic policies.
result Minimax estimator avoids double sampling issue and can incorporate off-policy data.
MERL uses evolutionary and gradient-based methods to optimize sparse team-based and dense agent-specific rewards in multiagent coordination.
problem Training multiagent reinforcement learning policies on sparse team-based rewards is difficult and relying solely on agent-specific rewards is sub-optimal.
method MERL employs a split-level training platform with an evolutionary algorithm and a gradient-based optimizer, transferring skills between the two processes.
result MERL significantly outperforms state-of-the-art methods on coordination benchmarks.
Variational Proximal Policy Optimization improves reinforcement learning from human feedback.
problem Policy mode collapse and brittle exploration loops in reinforcement learning.
method Particle-based variational inference framework with Mixture-of-Experts architecture.
result Significant improvements in complex reasoning benchmarks.
This research enhances exploration in DDPG using latent trajectory optimization.
problem Limited exploration in DDPG with deterministic policies.
method Model-based trajectory optimization for exploration in DDPG, using a learned deep dynamics model.
result Improved performance in continuous control tasks, especially with sparse rewards and images.
Paper proposes MCTSPO for better reinforcement learning policy optimization.
problem Local optima and saddle points in gradient-based methods and poor initialization in gradient-free methods.
method Monte-Carlo tree search combined with gradient-free optimization.
result Improved performance on reinforcement learning tasks with deceptive or sparse reward functions.
Sparse RSP routing improves graph exploration and classification.
problem Optimal randomized routing and distance measures on weighted graphs.
method Tsallis divergence regularization for sparse RSP.
result Sparse random walk converges to least-cost graph as temperature decreases.
BRPO optimizes batch RL policies to better exploit state-action differences.
problem Batch RL's conservatism limits exploitation of state-action differences.
method Proposes residual policies and derives BRPO to maximize policy performance.
result BRPO achieves state-of-the-art performance in various tasks.
MPPO improves exploration efficiency for on-policy methods.
problem Inefficient exploration in on-policy methods.
method MPPO uses a population of diverse policies to enable better exploration.
result MPPO significantly outperforms state-of-the-art exploration methods.
We propose a sequential learning policy for noisy discrete global optimization and ranking and selection (R\&S) problems with high dimensional sparse belief functions, where there are hundreds or even thousands of features, but only a small portion of these features contain explanatory power. We aim to identify the spa…
The paper tackles reward-relevance in offline RL with sparse decision dynamics.
problem Offline reinforcement learning with sparse decision dynamics and estimation sparsity.
method Reward-filtered least-squares policy evaluation using thresholded lasso.
result The method provides theoretical guarantees with sample complexity dependent on sparse component size.
SAC-NF improves RL exploration by using normalizing flows to discover policies faster.
problem Discovering optimal policies in sparse reward domains.
method Extending SAC with normalizing flows to improve exploration efficiency.
result SAC-NF accelerates policy discovery with smaller parameter usage.
SAGE enhances reinforcement learning by injecting hints to prevent model stagnation.
problem Sparse rewards cause large language models to stall under relative policy optimization.
method SAGE injects privileged hints during training to increase within-group outcome diversity.
result SAGE consistently outperforms GRPO on 6 benchmarks with LLMs, achieving significant improvements.
Develops HMRL for sparse reward RL problems, improving meta policy efficiency and transferability.
problem Difficulty in learning meta policies for sparse reward RL problems.
method Hyper-Meta RL framework with cross-environment meta state embedding and shaped meta reward.
result Improves meta policy generalization and efficiency for sparse reward RL problems.
This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.
problem Lack of exploration in state space when maximizing policy entropy.
method Proposes maximizing the entropy of a lower bound approximation to the state weighting distribution, based on latent space representation.
result Entropy regularization based on marginal state distribution achieves superior state space coverage and better performance in various domains.
New RL method uses distance between states instead of rewards for sparse reward environments.
problem Sparse rewards or non-reward environments in reinforcement learning.
method Uses goal-distance gradient and bridge point planning for policy improvement.
result Significantly better performance on sparse reward and local optimal problems in complex environments.
MeRL learns from sparse, underspecified rewards by discounting spurious trajectories.
problem Learning from binary success-failure feedback with little context.
method MeRL uses KL divergence to collect diverse successful trajectories and optimize an auxiliary reward function.
result MeRL outperforms alternative reward learning techniques and achieves state-of-the-art performance.
New algorithms reduce sample complexity for multiclass contextual bandits.
problem Designing efficient algorithms for multiclass contextual bandits with sparse rewards.
method Two complementary approaches: decision-estimation coefficient analysis and low-variance exploration.
result Achieved optimal sample complexity bounds for multiclass contextual bandits.
NADPEx uses dropout to enable temporally consistent exploration in reinforcement learning.
problem Achieving temporally consistent exploration in reinforcement learning agents.
method Integrates dropout into reinforcement learning policies to ensure temporal consistency.
result NADPEx outperforms naive exploration and parameter noise in tasks with sparse rewards.
Sparse representations improve reinforcement learning control policies.
problem Sparse representations are underused in reinforcement learning control.
method Incremental learning with sparse representations from neural networks, using distributional regularizers.
result Sparse representations avoid catastrophic interference and provide stable values for reinforcement learning.
Enhances reinforcement learning with hierarchical policies using latent variables.
problem Improving performance in reinforcement learning tasks with hierarchical policies.
method Training each layer of a hierarchical neural network to solve tasks directly, with latent variables controlling lower layers.
result Improves performance on standard benchmark tasks and complex sparse-reward tasks.
LEARN-SAM improves RL from sub-optimal demonstrations by localizing expert policies and selectively using demonstrations.
problem Improving RL from sub-optimal or sparse demonstrations.
method Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM).
result LEARN-SAM boosts learning speed and accuracy by selectively using demonstrations.
A new algorithm RTO integrates DPO and PPO for RLHF, improving performance.
problem Learning from sparse, sentence-level rewards in RLHF.
method Modeling RLHF as MDP, introducing RTO that learns token-wise reward function.
result RTO outperforms PPO on benchmarks by 7.5 points on AlpacaEval 2 and 4.1 points on Arena-Hard.
QAM uses adjoint matching to optimize continuous-action RL policies efficiently.
problem Efficient optimization of expressive diffusion or flow-matching policies with respect to a Q-function.
method QAM leverages adjoint matching to bypass the numerical instability of backpropagation through multi-step denoising processes.
result QAM consistently outperforms prior approaches on hard, sparse reward tasks in offline and offline-to-online RL.
SAIL learns from sub-optimal demonstrations to improve sample efficiency in sparse reward tasks.
problem Reducing sample complexity in sparse-rewarded tasks.
method Self-Adaptive Imitation Learning (SAIL) that exploits sub-optimal demonstrations and efficient exploration.
result Significantly improved sample efficiency and better final performance across various tasks.
ESPD improves learning efficiency in sparse reward reinforcement learning.
problem Sparse reward reinforcement learning challenges.
method Evolutionary Stochastic Policy Distillation (ESPD) based on drifted random walk insight.
result High learning efficiency demonstrated in MuJoCo robotics control suite experiments.
This thesis tackles policy learning in large action spaces using on-policy and off-policy methods.
problem Challenges in learning with large action spaces, including inefficient exploration, sparse data, high-variance importance weights, and difficult optimization.
method Developed structured Bayesian methods for on-policy learning and structured direct methods for off-policy learning.
result Introduced meTS and dTS for on-policy learning, and sDM for off-policy learning, with regret guarantees and optimization strategies.
Active learning selects optimal measurement times for inferring continuous paths from sparse data.
problem Inferring continuous probability paths from sparse snapshots in high-fidelity domains like single-cell biology.
method Extends active experimentation to the space of measures using Linearized Optimal Transport (LOT) for probabilistic surrogate modeling.
result Empirical results show that the proposed strategy outperforms uncertainty-agnostic baselines.
A new EM framework for goal-conditioned RL improves performance on sparse reward tasks.
problem Handling sparse rewards in goal-conditioned reinforcement learning.
method A graphical model framework with an EM algorithm that includes a learning-in-hindsight E-step and a supervised M-step.
result hEM significantly outperforms model-free baselines on goal-conditioned benchmarks with sparse rewards.
Policy-GNN optimizes GNN aggregation for diverse node iterations.
problem Optimizing GNN performance by varying aggregation iterations for different nodes.
method Policy-GNN uses a meta-policy framework with deep reinforcement learning to adaptively determine the number of aggregations for each node.
result Policy-GNN significantly outperforms state-of-the-art alternatives on real-world datasets.
MAPO uses a memory buffer to improve policy optimization in structured prediction tasks.
problem Improving sample efficiency and robustness in policy optimization for structured prediction tasks.
method Memory Augmented Policy Optimization (MAPO) uses a memory buffer to reduce policy gradient variance.
result MAPO achieves state-of-the-art results in program synthesis and semantic parsing tasks.
Proposes MCTE framework for efficient imitation learning.
problem Efficiently learn sparse multi-modal policies from demonstrations.
method Maximum causal Tsallis entropy (MCTE) framework with sparsemax distribution.
result MCTEIL algorithm outperforms existing methods in simulations.
This work improves Hindsight Learning for goal-directed tasks in reinforcement learning.
problem Sparse reward problems in reinforcement learning, especially goal-directed tasks.
method Improves Hindsight Experience Replay and Hindsight Policy Gradients for robotic tasks.
result Improved learning in Hindsight Experience Replay and Hindsight Policy Gradients.