BASIS improves LLM reasoning by sharing batchwise rollout info, reducing MSE by 69%.
problem Improving large language model reasoning with limited rollouts and batch information.
method BASIS samples only one rollout per prompt but uses batch information to improve value function estimation.
result BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++.
Efficiently reduces computational burden of rollout acquisition functions in Bayesian optimization.
problem Expensive computation of rollout acquisition functions in Bayesian optimization.
method Combines quasi-Monte Carlo, common random numbers, and control variates to reduce computational burden. Formulates a policy-search approach to eliminate the need to optimize the rollout acquisition function.
result Significant reduction in computational burden of rollout acquisition functions.
Study on indexability of restless multi-armed bandits and rollout policy performance.
problem Maximizing discounted rewards in finite state restless multi-armed bandit problems.
method Decouple the problem into single-armed restless bandits, analyze using value iteration, and compare with Whittle index policy.
result Demonstrates conditions for indexability and compares performance of index policy and rollout policy.
SFPO optimizes LLM reasoning by repositioning before updating, improving stability and efficiency.
problem Noisy gradients from low-quality rollouts cause instability and inefficient exploration in on-policy RL algorithms.
method Decomposes each step into three stages: a short fast trajectory, repositioning, and slow correction, preserving the objective and rollout process unchanged.
result SFPO consistently improves stability, reduces rollouts, and accelerates convergence, outperforming GRPO on math reasoning benchmarks.
Several approximate policy iteration schemes without value functions, which focus on policy representation using classifiers and address policy learning as a supervised learning problem, have been proposed recently. Finding good policies with such methods requires not only an appropriate classifier, but also reliable e…
A new estimator combines bootstrapping and rollout methods in RL.
problem Combining strengths of bootstrapping and rollout methods in RL.
method Subgraph Bellman operators and fixed point solving.
result Upper bound on error approaches optimal TD variance with additional term.
This paper optimizes sampling policies for Bayesian optimization to improve exploration and exploitation.
problem Improving the balance between exploration and exploitation in Bayesian optimization.
method Developed efficient methods to estimate and optimize non-myopic acquisition functions using rollout policies and stochastic gradient optimization.
result Efficient optimization of sampling policies leads to better performance in Bayesian optimization.
There are two halves to RL systems: experience collection time and policy learning time. For a large number of samples in rollouts, experience collection time is the major bottleneck. Thus, it is necessary to speed up the rollout generation time with multi-process architecture support. Our work, dubbed WALL-E, utilizes…
AIS corrects rollout-training mismatch in quantized RL, improving speed and stability.
problem Rollout-training mismatch in quantized RL causes bias and training collapse.
method Adaptive Importance Sampling (AIS) adjusts gradient correction per batch.
result AIS matches BF16 baseline on most tasks while improving speed.
Sequence generation models are commonly refined with reinforcement learning over user-defined metrics. However, high gradient variance hinders the practical use of this method. To stabilize this method, we adapt to contextual generation of categorical sequences a policy gradient estimator, which evaluates a set of corr…
sGPO optimizes RLVR training by balancing inference FLOPs and training efficiency.
problem RLVR training allocates rollout budget without considering query difficulty.
method sGPO trades inference FLOPs for reduced training FLOPs.
result sGPO matches or exceeds baseline performance while reducing training compute by a factor of three.
CoDistill-GRPO improves small models in GRPO by distilling knowledge from a larger model.
problem Small models in GRPO struggle with sparse rewards on difficult tasks.
method Simultaneously trains a large and small model using co-distillation and GRPO objectives.
result Significant improvement in small model performance over standard GRPO on mathematical benchmarks.
Lookahead, also known as non-myopic, Bayesian optimization (BO) aims to find optimal sampling policies through solving a dynamic program (DP) that maximizes a long-term reward over a rolling horizon. Though promising, lookahead BO faces the risk of error propagation through its increased dependence on a possibly mis-sp…
UK's rapid vaccine rollout linked to reduced COVID-19 mortality.
problem Assessing the impact of accelerated vaccine rollout on public health outcomes.
method Flexible probabilistic models combining interrupted time series analysis and synthetic control methods with multi-output Gaussian processes.
result Substantial reduction in COVID-19 mortality with little effect on transmission rates.
A new method reduces compounding errors in model-based reinforcement learning.
problem Compounding errors in long horizon predictions from model-based reinforcement learning.
method Maximum Entropy Model Rollouts (MEMR) with non-uniform sampling and prioritized experience replay.
result Significantly reduces computation requirements compared to other model-based methods.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
InfoTree improves reinforcement learning by optimizing tool use with a greedy submodular approach.
problem Maximizing information from tool use in reinforcement learning with limited resources.
method Formalizes Rollout Informativeness, recasts state selection as submodular maximization, and uses UUCB and ABA.
result InfoTree outperforms existing methods across various benchmarks, improving performance by 18.2% on average.
Improves data efficiency in multi-agent control tasks using model-based reinforcement learning.
problem Limited data efficiency in reinforcement learning for multi-agent tasks.
method Decentralized model-based policy optimization (DMPO) framework.
result DMPO achieves superior data efficiency and matches model-free methods using true models.
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…
SAGE enhances reinforcement learning by injecting hints to prevent model stagnation.
problem Sparse rewards cause large language models to stall under relative policy optimization.
method SAGE injects privileged hints during training to increase within-group outcome diversity.
result SAGE consistently outperforms GRPO on 6 benchmarks with LLMs, achieving significant improvements.
New method optimizes language model performance for test-time strategies.
problem Mismatch between training objectives and test-time deployment of large language models.
method Tail-Extrapolated estimators to approximate best-of-N performance from limited training rollouts.
result Improved performance of best-of-N deployment across various models and datasets.
This paper proposes a novel deep reinforcement learning architecture that was inspired by previous tree structured architectures which were only useable in discrete action spaces. Policy Prediction Network offers a way to improve sample complexity and performance on continuous control problems in exchange for extra com…
In this paper we target the problem of transferring policies across multiple environments with different dynamics parameters and motor noise variations, by introducing a framework that decouples the processes of policy learning and system identification. Efficiently transferring learned policies to an unknown environme…
Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining re…
New framework for learning policies that converge in out-of-sample regions.
problem Reliable out-of-sample recovery in imitation learning.
method Contractive dynamical systems and recurrent equilibrium networks.
result Policy rollouts converge regardless of perturbations, enabling efficient OOS recovery.
Researchers use DT to transfer policies from one environment to another using causal reasoning.
problem Adapting to changes in environmental dynamics in reinforcement learning.
method Applying causal counterfactual reasoning to Decision Transformer (DT) architecture for policy transfer.
result DT successfully transfers a learned policy to new environments while retaining most of the reward.
Unified framework for reliable uncertainty quantification in RL.
problem Uncertainty quantification in high-stakes reinforcement learning.
method Unified conformal prediction framework integrating distributional RL and conformal calibration.
result Significantly improved coverage and reliability over standard methods.
Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neura…
Paper proposes BMPO to optimize policies using bidirectional models.
problem Model-based reinforcement learning's reliance on forward model accuracy.
method Develops BMPO using both forward and backward models for policy optimization.
result BMPO outperforms state-of-the-art methods in sample efficiency and asymptotic performance.
Generative model learns to mimic time-series behavior to generate accurate trajectories.
problem Learning generative models for time-series data with accurate multi-step trajectories.
method Contrastive imitation framework combining autoregressive and adversarial elements.
result The method generates accurate and useful samples from real-world datasets.
ZDPG learns model-free policies without critics, improving on PG.
problem Model-free policy learning in complex dynamic problems.
method Approximates policy-reward gradients via two-point stochastic evaluations of the Q-function.
result Restores true model-free policy learning without critics, with improved stability and efficiency.
Paper proposes a forecasting solution for network-rollout planning.
problem Accurate estimation of milestone completion times for network-rollout planning.
method Partition-based regression models incorporating data-driven statistical models.
result The proposed approach outperforms alternative models in terms of performance and model complexity.
ReSkill reconciles RL skill creation with policy optimization.
problem RL policies lack reusable strategies across tasks.
method Integrates skill creation into RL loop with three mechanisms.
result Consistently outperforms existing methods, especially on unseen tasks.
State entropy regularization improves robustness in reinforcement learning, especially under structured perturbations.
problem Structured and spatially correlated perturbations in reinforcement learning.
method State entropy regularization, compared to policy entropy.
result State entropy regularization provides better robustness to structured and spatially correlated perturbations.
Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-b…
Monte Carlo Tree Search (MCTS) algorithms have achieved great success on many challenging benchmarks (e.g., Computer Go). However, they generally require a large number of rollouts, making their applications costly. Furthermore, it is also extremely challenging to parallelize MCTS due to its inherent sequential nature:…
A contraction analysis improves model-based RL's error recovery.
problem Theoretical understanding of model-based reinforcement learning.
method Contraction analysis applied to both stochastic and deterministic state transitions.
result Error reduction in cumulative reward using branched rollouts.
Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexity. In this paper we consider the deterministic value gradients to improve the sample efficiency of …
New RL algorithm explains why deep learning works in stochastic environments.
problem Why deep RL algorithms perform well in practice despite using random exploration.
method Introducing SQIRL, an iterative RL algorithm that separates exploration and learning.
result Effective horizon explains why deep RL works in stochastic environments.
A new method for learning policies from demonstrations without reinforcement.
problem Learning policies from demonstrations without access to reinforcement signals.
method Energy-based distribution matching (EDM) to learn policy parameters and state marginals.
result EDM yields consistent performance gains over existing algorithms for strictly batch imitation learning.
NM-PPG optimizes adaptive feature acquisition in POMDPs for better predictions.
problem Optimizing adaptive feature acquisition in prediction problems with costly features.
method Non-myopic pathwise policy gradients (NM-PPG) with continuous relaxation and straight-through rollout.
result NM-PPG outperforms state-of-the-art AFA methods on synthetic and real-world datasets.
Deep learning solves EV routing with time windows for EV fleets.
problem Electric vehicle routing with time windows for logistics.
method End-to-end deep reinforcement learning with attention model and graph embedding.
result Proposed model efficiently solves large EVRPTW instances.
A framework for navigating environments with spatially correlated obstacles and uncertain blockage status.
problem Navigation in environments with spatially correlated obstacles of uncertain blockage status.
method Modeling spatial correlation with Gaussian Random Field, developing Bayesian belief updates, proposing a two-stage learning framework with offline and online phases.
result Consistent performance gains over baselines in environments with adversarial interruptions or clustered natural hazards.
Optimizes trading policies using future price forecasts.
problem Static reinforcement learning agents lack mechanisms for using price forecasts at inference time.
method FPILOT framework inspired by Model Predictive Control (MPC). Uses a predictive model to construct an allocation-based imagined return objective at each decision step.
result Consistent improvements in total return and risk-adjusted metrics across various policy learning algorithms.
Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience …
The decision to rollout a vehicle is critical to fleet management companies as wrong decisions can lead to additional cost of maintenance and failures during journey. With the availability of large amount of data and advancement of machine learning techniques, the rollout decisions of a supervisor can be effectively au…
Optimal sample complexity for autoregressive chain-of-thought learning proven.
problem Determining the minimum number of samples needed for accurate autoregressive chain-of-thought learning.
method Proved upper bound on sample complexity using Daniely-Shalev-Shwartz dimension and roll-out stable parity dimension.
result The sample complexity is bounded by the local next-token class rate, with no dependence on rollout length.
Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that…