A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optim…
RRPI improves offline RL by optimizing policies against worst-case dynamics.
Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks. The regularized formulation modifies the standard RL objective and thus generally converges to a policy different from the optimal gree…
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at t…
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
Policy iteration is a family of algorithms that are used to find an optimal policy for a given Markov Decision Problem (MDP). Simple Policy iteration (SPI) is a type of policy iteration where the strategy is to change the policy at exactly one improvable state at every step. Melekopoglou and Condon [1990] showed an exp…
A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or -function may fail to improve performance---or worse, actually cause the policy performance …
New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.
Accelerates policy optimization in RL with optimistic and adaptive updates.
One-step policy improvement outperforms iterative RL methods on D4RL.
The paper tackles robust reinforcement learning with performance guarantees.
KL-constrained API shows optimization issues and improved with regularization.
Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and…
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…
Policy-GNN optimizes GNN aggregation for diverse node iterations.
Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this paper, we draw connections between DP and (constrained) convex optimization. Specifically, we show clear links in the algorithmic structure b…
Improved API to achieve optimal error bound and query complexity in local planning.
We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…
Mixed RL improves RL efficiency with dual representations.
Lower bounds for PI on multi-action MDPs are established, showing complexity grows with action count.
MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.
Paper tackles offline SSP with value iteration for policy evaluation and learning.
We consider the infinite-horizon discounted optimal control problem formalized by Markov Decision Processes. We focus on several approximate variations of the Policy Iteration algorithm: Approximate Policy Iteration, Conservative Policy Iteration (CPI), a natural adaptation of the Policy Search by Dynamic Programming a…
Policy gradient methods achieve linear convergence in simple MDPs.
Efficient local planning with linear approximations for agents with limited simulator access.
This paper addresses the model-free nonlinear optimal problem with generalized cost functional, and a data-based reinforcement learning technique is developed. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton-Jacobi-Bellman (HJB) equation, which is a nonlinear partial differ…
Single-timescale actor-critic finds globally optimal policy.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
Adaptive optimal control using value iteration initiated from a stabilizing control policy is theoretically analyzed in terms of stability of the system during the learning stage without ignoring the effects of approximation errors. This analysis includes the system operated using any single/constant resulting control …
Paper uses RL to optimize multi-asset portfolios in fluctuating markets.
In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…
In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation erro…
New algorithm learns optimal policies without explicit rewards.
Develops a reinforcement learning algorithm for learning deterministic equilibrium policies in time-inconsistent control problems.
LF-IBIS learns optimal policies online without explicit likelihood.
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
The paper analyzes optimal liquidation strategies for cryptocurrencies considering both temporary and permanent price impacts.
New algorithm finds near-optimal policies efficiently in zero-sum games.
Optimizes trading strategy considering alpha decay and transaction costs.
Softmax PG methods can take extremely long to converge, even with exact gradients.
Paper analyzes NAC with neural networks for efficient policy optimization.
Active search is a learning paradigm for actively identifying as many members of a given class as possible. A critical target scenario is high-throughput screening for scientific discovery, such as drug or materials discovery. In this paper, we approach this problem in Bayesian decision framework. We first derive the B…
In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based…
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraints, normalizing flows policy generates samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps av…
Policy gradient converges linearly with Hadamard parameterization in tabular settings.
Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspi…