The paper proposes an iterative approach to batch reinforcement learning for safer and more informative data collection.
problem Learning policies that are too rigid and do not adapt to new data.
method Safe diversified model-based policy search in an iterative batch reinforcement learning framework.
result Improved learned policies through continuous data collection and adaptation.
The paper explores how duality applies to reinforcement learning.
problem Exploring the use of duality in reinforcement learning.
method Demonstrates duality in various RL methods, including value iteration and approximations.
result Lagrangian duality is found to be prevalent in reinforcement learning.
Graph neural network executes value iteration for flexible environments.
problem Value iteration in flexible environments with direct supervision.
method Graph Neural Network (GNN) executing value iteration algorithm.
result GNN accurately models value iteration across diverse environments.
Paper improves worst-case regret bounds for RLSVI in reinforcement learning.
problem Minimizing regret in reinforcement learning with randomized value functions.
method Introduces a clipping variant of Thompson Sampling for RLSVI.
result Achieves a i l d e O ( H 2 S A T ) ilde{\mathrm{O}}(H^2S\sqrt{AT}) i l d e O ( H 2 S A T ) worst-case regret bound. Deep RL agents suffer from transient non-stationarity, which ITER mitigates.
problem Transient non-stationarity in deep RL agents affects generalization.
method Iterated Relearning (ITER) transfers knowledge between networks to reduce non-stationarity.
result ITER improves deep RL agents' performance on generalization benchmarks.
We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the mode…
The paper tackles robust reinforcement learning with performance guarantees.
problem Finding a robust policy for RMDP with state space uncertainties.
method Proposes RLSPI algorithm for learning optimal robust policy with performance bounds.
result Demonstrates the performance of RLSPI on standard benchmark problems.
XLVINs improve deep reinforcement learning by combining self-supervised learning and neural algorithmic reasoning.
problem Limitations of Value Iteration Networks (VINs) in deep reinforcement learning.
method Combining contrastive self-supervised learning, graph representation learning, and neural algorithmic reasoning.
result XLVINs match VIN-like models on discrete, fixed MDPs and significantly outperform model-free baselines.
We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive q q q -functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …
Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper w…
MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.
problem Optimizing risk in reinforcement learning control problems.
method Mean-Variance Policy Iteration (MVPI) framework for risk-averse control in MDPs.
result Risk-averse TD3 outperforms previous methods in robot simulation tasks.
A reinforcement learning method predicts medical outcomes from masked feature vectors.
problem Predicting medical outcomes from limited feature data.
method Reinforcement learning with iterative feature selection and a guesser network.
result The method outperforms baselines and is highly interpretable.
RLCFR improves CFR's generalization in imperfect information games.
problem Improving CFR's performance in large-scale, imperfect information games.
method RLCFR integrates CFR with deep reinforcement learning to update strategies dynamically.
result RLCFR significantly enhances CFR's generalization ability in various games.
Paper uses RL to optimize multi-asset portfolios in fluctuating markets.
problem Optimizing multi-asset portfolios in time-varying financial markets.
method Soft Actor-Critic (SAC) algorithm for policy learning, policy iteration process.
result SAC algorithm outperforms in various criteria in simulated and real financial markets.
The paper reclassifies RL algorithms using inference concepts.
problem To provide a unified perspective on RL algorithms.
method Using the control as inference framework, the paper classifies RL algorithms as amortised and iterative inference.
result A wide range of RL algorithms can be classified in this new manner.
Sublinear LSVI via LSH reduces runtime to sublinear in actions.
problem Efficiently estimating value functions in reinforcement learning with sublinear runtime.
method Formulated as approximate maximum inner product search, used LSH to solve with sublinear time complexity.
result Sublinear runtime while maintaining LSVI's regret.
MaxEnt Model Correction improves reinforcement learning model accuracy.
problem Improving reinforcement learning model accuracy and convergence.
method MaxEnt Model Correction (MoCo) procedure to correct model's next-state distributions.
result MoCoVI and MoCoDyna algorithms converge faster and effectively use approximate models.
Unified reinforcement learning methods using hybrid inference.
problem Combining model-based and model-free reinforcement learning approaches.
method Control as Hybrid Inference (CHI) framework.
result CHI algorithm balances model-based and model-free learning.
A framework for reinforcement learning tackles CVRP with competitive results.
problem Optimizing routes for vehicles with limited capacity.
method Formulates action selection as a mixed-integer optimization problem, uses policy iteration to improve policies.
result Achieves an average gap of 1.7% with state-of-the-art OR methods on CVRP instances.
Proof shows imitation of expert's reward and solutions in multi-objective optimization.
problem Multi-objective optimization with reward and solution imitation.
method Wasserstein inverse reinforcement learning.
result Wasserstein inverse reinforcement learning enables imitation of expert's reward and solutions in multi-objective optimization.
MaxMax Q-Learning improves coordination in multi-agent reinforcement learning by refining action selection.
problem Relative over-generalization in decentralized multi-agent reinforcement learning.
method MaxMax Q-Learning employs iterative sampling and evaluation of potential next states to refine approximations of ideal state transitions.
result MaxMax Q-Learning frequently outperforms existing baselines, demonstrating enhanced convergence and sample efficiency.
DAC enhances exploration in reinforcement learning with entropy regularization.
problem Improving exploration efficiency in reinforcement learning.
method Sample-aware entropy regularization using replay buffer action distributions.
result DAC significantly outperforms existing algorithms in reinforcement learning tasks.
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
A new algorithm uses IVs to learn optimal policies from observational data.
problem Learning optimal policies from unobserved variable confounded data.
method IV-aided Value Iteration (IVVI) algorithm based on conditional moment restrictions.
result First provably efficient algorithm for instrument-aided offline RL.
Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.
problem Large linear systems in portfolio optimization and option pricing lead to slow convergence.
method Reinforcement Learning (RL) dynamically adjusts block-preconditioner sizes to accelerate convergence.
result RL-driven solver significantly reduces computational cost and accelerates convergence.
New algorithms accelerate value function convergence in MDPs.
problem Accelerating convergence of value functions in Markov Decision Processes (MDPs).
method Operator Splitting Value Iteration (OS-VI) and OS-Dyna.
result Achieves much faster convergence rate with accurate models.
OptEx accelerates first-order optimization with parallelized iterations.
problem Inefficiencies in first-order optimization algorithms for complex tasks.
method Approximately parallelized iterations using kernelized gradient estimation.
result OptEx achieves substantial efficiency improvements with an effective acceleration rate of Ω ( N ) Ω(\sqrt{N}) Ω ( N ) . Accelerates TD learning for long-horizon reinforcement learning problems.
problem Slow convergence of conventional TD learning in long-horizon tasks.
method Introduces PID Accelerated Temporal Difference (PID TD) learning algorithms.
result Accelerates convergence of TD learning compared to conventional methods.
New algorithm identifies near-optimal policies in adversarial distributed RL settings.
problem Adversarial agents in distributed RL settings that can collude and report arbitrary data.
method Weighted-Clique algorithm for robust mean estimation from batches, combined with novel distributed algorithms.
result Achieves superior robustness guarantees and near-optimal sample complexities in both offline and online settings.
Unified analysis of stochastic iterative algorithms using Lyapunov functions.
problem Analyzing convergence of stochastic iterative algorithms for fixed-point equations.
method Lyapunov-based techniques for finite-time analysis of stochastic approximation algorithms.
result Unified mean-square convergence guarantees for various algorithms.
Polynomial-time RL algorithm for constant actions under linear Bellman completeness.
problem Efficient online reinforcement learning with few actions.
method Polynomial-time algorithm based on linear function approximation.
result First computationally efficient algorithm for RL with constant actions under linear Bellman completeness.
LF-IBIS learns optimal policies online without explicit likelihood.
problem Bayesian RL challenges due to intractable likelihood functions.
method Combines ABC with IBIS for online belief updates.
result Approximates posterior distributions for policies and parameters.
Develops a reinforcement learning algorithm for learning deterministic equilibrium policies in time-inconsistent control problems.
problem Learning equilibrium policies in time-inconsistent control problems.
method Continuous-time model-free reinforcement learning algorithm using deterministic policy gradient approach.
result Learned equilibrium policies in general time-inconsistent control problems.
We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of least-squares value iteration that use Boltzmann or epsilon-greedy exploration can be highly ineffic…
Papers learn from data to make decisions without interacting, improving on previous methods.
problem Achieving optimal decision-making from offline data with non-linear function approximation.
method Pessimistic Nonlinear Least-Square Value Iteration (PNLSVI) with three innovative components.
result Achieves minimax optimal instance-dependent regret for non-linear function approximation.
Two new algorithms improve Q* approximation in batch RL with linear error propagation.
problem Improving Q* approximation in batch reinforcement learning.
method Two novel algorithms that estimate Bellman error directly, without quadratic dependence.
result Linear-in-horizon error propagation for batch RL algorithms.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
The performance of deep (reinforcement) learning systems crucially depends on the choice of hyperparameters. Their tuning is notoriously expensive, typically requiring an iterative training process to run for numerous steps to convergence. Traditional tuning algorithms only consider the final performance of hyperparame…
Mixed RL improves RL efficiency with dual representations.
problem Poor sampling efficiency in RL methods.
method Uses dual representations of environmental dynamics to search optimal policies.
result Proves convergence and recursive stability of the mixed RL.
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the mod…
Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement learning. In this paper, we explain briefly what Anderson acceleration is and how it can be applied to…
New RL algorithms improve average-reward performance.
problem Improving RL algorithms for average-reward criteria.
method Developed novel algorithms addressing average-reward criterion directly.
result ATRPO significantly outperforms TRPO in challenging MuJuCo environments.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this paper, we draw connections between DP and (constrained) convex optimization. Specifically, we show clear links in the algorithmic structure b…
IDS improves RLHF by smoothing reward data, enhancing model performance.
problem Reward model performance degrades and overoptimization hinders true objective.
method Iterative Data Smoothing (IDS) updates model and data labels during each epoch.
result IDS outperforms traditional methods in RLHF.
New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.
problem Iterative RLHF leads to alignment collapse where policies exploit RM's blind spots.
method Foresighted policy optimization (FPO) restores missing steering term via regularization.
result FPO prevents alignment collapse on LLM alignment pipelines using Llama-3.2-1B.