arXiv research

Value iteration is a fixed point iteration technique utilized to obtain the optimal value function and policy in a discounted reward Markov Decision Process (MDP). Here, a contraction operator is constructed and applied repeatedly to arrive at the optimal solution. Value iteration is a first order method and therefore …

Graph neural network executes value iteration for flexible environments.

XLVINs improve data efficiency in implicit planning by leveraging latent space.

New algorithms accelerate value function convergence in MDPs.

Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper w…

This paper presents a way of solving Markov Decision Processes that combines state abstraction and temporal abstraction. Specifically, we combine state aggregation with the options framework and demonstrate that they work well together and indeed it is only after one combines the two that the full benefit of each is re…

Novel algorithm for Markov decision processes using rank-one approximation.

Paper improves worst-case regret bounds for RLSVI in reinforcement learning.

In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…

New convergence rates for shuffling gradient methods without strong convexity.

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive $q$-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates …

Sublinear LSVI via LSH reduces runtime to sublinear in actions.

Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.

We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiab…

FVI method calculates bicausal OT with neural networks, outperforming other methods.

XLVINs improve deep reinforcement learning by combining self-supervised learning and neural algorithmic reasoning.

We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…

We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of least-squares value iteration that use Boltzmann or epsilon-greedy exploration can be highly ineffic…

The paper explores how duality applies to reinforcement learning.

Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.

In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…

Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optim…

Recursive stochastic algorithms have gained significant attention in the recent past due to data driven applications. Examples include stochastic gradient descent for solving large-scale optimization problems and empirical dynamic programming algorithms for solving Markov decision problems. These recursive stochastic a…

New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.

Papers learn from data to make decisions without interacting, improving on previous methods.

Polynomial-time RL algorithm for constant actions under linear Bellman completeness.

New algorithm learns optimal policies without explicit rewards.

We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the mode…

Proposes a method for obtaining interval bounds in off-policy evaluation.

Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these tw…

This work proposes a non-iterative strategy for missing value imputations which is guided by similarity between observations, but instead of explicitly determining distances or nearest neighbors, it assigns observations to overlapping buckets through recursive semi-random hyperplane cuts, in which weighted averages are…

We give a formula of the Upsilon invariant of any L-space cable knot $K_{p,q}$ using $p,Υ_K$ and $Υ_{T_{p,q}}$. The integral value of the Upsilon invariant gives a ${\mathbb Q}$-valued knot concordance invariant. We compute the integral values for L-space iterated cable knots.

New algorithm identifies near-optimal policies in adversarial distributed RL settings.

A new algorithm uses IVs to learn optimal policies from observational data.

Policy gradient method proves convergence in imperfect-information games.

Paper tackles offline SSP with value iteration for policy evaluation and learning.

A new method calibrates value predictions in offline RL to improve reliability.

This study is aimed at answering the famous question of how the approximation errors at each iteration of Approximate Dynamic Programming (ADP) affect the quality of the final results considering the fact that errors at each iteration affect the next iteration. To this goal, convergence of Value Iteration scheme of ADP…

MaxEnt Model Correction improves reinforcement learning model accuracy.

A new imputation method considers feature importance for missing value completion.

Adaptive optimal control using value iteration initiated from a stabilizing control policy is theoretically analyzed in terms of stability of the system during the learning stage without ignoring the effects of approximation errors. This analysis includes the system operated using any single/constant resulting control …

Following the lines of a celebrated result by R. Bott (Comm. Pure Appl. Math. 9, 1956) we study the Morse index of the iterated of a closed geodesic in stationary Lorentzian manifolds, or, more generally, of a closed Lorentzian geodesic that admits a timelike periodic Jacobi field. Given one such closed geodesic $γ$, w…

Solves capillary curvature problems for specific p values.

We propose a novel method for maximum likelihood-based parameter inference in nonlinear and/or non-Gaussian state space models. The method is an iterative procedure with three steps. At each iteration a particle filter is used to estimate the value of the log-likelihood function at the current parameter iterate. Using …

Study tight offline learning bounds for linear MDPs using variance information.

RRPI improves offline RL by optimizing policies against worst-case dynamics.

IDS improves RLHF by smoothing reward data, enhancing model performance.

KL regularization helps RL algorithms by implicitly averaging q-values.