A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
SIMPOL solves complex economic models using numerical methods.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
This paper addresses the model-free nonlinear optimal problem with generalized cost functional, and a data-based reinforcement learning technique is developed. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton-Jacobi-Bellman (HJB) equation, which is a nonlinear partial differ…
Paper tackles DOCTR-L with SciPhy RL, solving neural PDEs from data.
Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…
Revel tackles safe exploration in RL with verified symbolic policies.
New method for robust policy evaluation in offline reinforcement learning with sequentially exogenous unobserved confounders.
Graph neural network executes value iteration for flexible environments.
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
Policy iteration is a family of algorithms that are used to find an optimal policy for a given Markov Decision Problem (MDP). Simple Policy iteration (SPI) is a type of policy iteration where the strategy is to change the policy at exactly one improvable state at every step. Melekopoglou and Condon [1990] showed an exp…
Solves POMDPs with recurrent neural networks and natural policy gradient.
Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optim…
One-step policy improvement outperforms iterative RL methods on D4RL.
RRPI improves offline RL by optimizing policies against worst-case dynamics.
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Mark…
MoMA improves model-based RL by using unrestricted policy classes.
We consider the infinite-horizon discounted optimal control problem formalized by Markov Decision Processes. We focus on several approximate variations of the Policy Iteration algorithm: Approximate Policy Iteration, Conservative Policy Iteration (CPI), a natural adaptation of the Policy Search by Dynamic Programming a…
The paper proposes an iterative approach to batch reinforcement learning for safer and more informative data collection.
We formulate simple assumptions, implying the Robbins-Monro conditions for the -learning algorithm with the local learning rate, depending on the number of visits of a particular state-action pair (local clock) and the number of iteration (global clock). It is assumed that the Markov decision process is communicatin…
We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…
A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or -function may fail to improve performance---or worse, actually cause the policy performance …
New -step policy gradient method avoids local optima in restricted policy classes.
Policy gradient method proves convergence in imperfect-information games.
NeuPL learns diverse policies in strategy games efficiently.
Study optimal policy regret in partially observable Markov games with adaptive opponents.
Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at t…
New framework learns policies for partially observable systems.
The paper tackles robust reinforcement learning with performance guarantees.
New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.
Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks. The regularized formulation modifies the standard RL objective and thus generally converges to a policy different from the optimal gree…
In this paper, we prove some convergence results of a special case of optimistic policy iteration algorithm for stochastic shortest path problem. We consider both Monte Carlo and methods for the policy evaluation step under the condition that the termination state will eventually be reached almost surely.
Policy gradient is a generic and flexible reinforcement learning approach that generally enjoys simplicity in analysis, implementation, and deployment. In the last few decades, this approach has been extensively advanced for fully observable environments. In this paper, we generalize a variety of these advances to part…
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
Lower bounds for PI on multi-action MDPs are established, showing complexity grows with action count.
Efficient integration of uncertain observations with decision-making optimization is key for prescribing informed intervention actions, able to preserve structural safety of deteriorating engineering systems. To this end, it is necessary that scheduling of inspection and monitoring strategies be objectively performed o…
Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally …
Policy gradient methods achieve linear convergence in simple MDPs.
This work characterizes reward function partial identifiability and its impact on policy optimization.
In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation erro…
This work improves policy-based training by proposing an evaluation balance objective for GFlowNets.
The paper introduces a novel method for stable off-policy learning using value function chaining.
CPPO learns policies from partial offline data in MDPs with structural assumptions.
We provide non-asymptotic bounds for the well-known temporal difference learning algorithm TD(0) with linear function approximators. These include high-probability bounds as well as bounds in expectation. Our analysis suggests that a step-size inversely proportional to the number of iterations cannot guarantee optimal …
PRL improves off-policy evaluation in partially observed MDPs.
Efficient local planning with linear approximations for agents with limited simulator access.
A new algorithm trains experts to safely guide agents in partially observed environments.