Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper w…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “approximate value iteration”
Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.
problem Approximation errors in policy and value function approximations.
method State-aggregated representations and policy gradient methods.
result Policy gradient methods can achieve a per-period regret bounded by ε, while approximate policy iteration and value iteration have a higher regret.
Novel algorithm for Markov decision processes using rank-one approximation.
problem Solving planning and learning problems of Markov decision processes.
method Policy iteration with rank-one approximation of transition probability matrix.
result The proposed algorithm consistently outperforms first-order algorithms and their accelerated versions.
New algorithms accelerate value function convergence in MDPs.
problem Accelerating convergence of value functions in Markov Decision Processes (MDPs).
method Operator Splitting Value Iteration (OS-VI) and OS-Dyna.
result Achieves much faster convergence rate with accurate models.
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…
Sublinear LSVI via LSH reduces runtime to sublinear in actions.
problem Efficiently estimating value functions in reinforcement learning with sublinear runtime.
method Formulated as approximate maximum inner product search, used LSH to solve with sublinear time complexity.
result Sublinear runtime while maintaining LSVI's regret.
Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscs.LG
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
Improved RL algorithm reduces regret in large state spaces.
problem Exploration in large or continuous state spaces.
method Optimistically-initialized randomized least-squares value iteration (RLSVI) with function approximation.
result Frequentist regret bound of for low-rank transition dynamics.
Papers learn from data to make decisions without interacting, improving on previous methods.
problem Achieving optimal decision-making from offline data with non-linear function approximation.
method Pessimistic Nonlinear Least-Square Value Iteration (PNLSVI) with three innovative components.
result Achieves minimax optimal instance-dependent regret for non-linear function approximation.
FVI method calculates bicausal OT with neural networks, outperforming other methods.
problem Computing bicausal optimal transport with adapted coupling structures.
method FVI method using multilayer neural networks to approximate value functions.
result FVI method outperforms linear programming and Sinkhorn methods in scalability.
Theoretical and Numerical Analysis of Approximate Dynamic Programming with Approximation Errorseess.SY
This study is aimed at answering the famous question of how the approximation errors at each iteration of Approximate Dynamic Programming (ADP) affect the quality of the final results considering the fact that errors at each iteration affect the next iteration. To this goal, convergence of Value Iteration scheme of ADP…
Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actionscs.LG
Polynomial-time RL algorithm for constant actions under linear Bellman completeness.
problem Efficient online reinforcement learning with few actions.
method Polynomial-time algorithm based on linear function approximation.
result First computationally efficient algorithm for RL with constant actions under linear Bellman completeness.
This paper presents a way of solving Markov Decision Processes that combines state abstraction and temporal abstraction. Specifically, we combine state aggregation with the options framework and demonstrate that they work well together and indeed it is only after one combines the two that the full benefit of each is re…
Stability Analysis of Optimal Adaptive Control using Value Iteration with Approximation Errorsmath.OC
Adaptive optimal control using value iteration initiated from a stabilizing control policy is theoretically analyzed in terms of stability of the system during the learning stage without ignoring the effects of approximation errors. This analysis includes the system operated using any single/constant resulting control …
MaxEnt Model Correction improves reinforcement learning model accuracy.
problem Improving reinforcement learning model accuracy and convergence.
method MaxEnt Model Correction (MoCo) procedure to correct model's next-state distributions.
result MoCoVI and MoCoDyna algorithms converge faster and effectively use approximate models.
New method for optimistic planning in MDPs using regularization.
problem Optimistic planning in infinite-horizon discounted MDPs.
method Regularized dynamic programming for approximate value iteration.
result Achieves near-optimal statistical guarantees in learning policies.
SIM-Shapley improves SV approximation efficiency and stability.
problem High computational costs of Shapley value methods in high-dimensional settings.
method Stochastic Iterative Momentum for Shapley Value Approximation (SIM-Shapley).
result Reduced computation time by up to 85% while maintaining feature attribution quality.
We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiab…
Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control pro…
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the mod…
AAPI improves regret bound for undiscounted continuing learning in uniformly ergodic MDPs.
problem Improving regret bounds for undiscounted continuing learning in uniformly ergodic MDPs.
method Adaptive approximate policy iteration (AAPI) with online learning techniques and data-dependent adaptive learning rate.
result AAPI achieves a regret bound, improving over the best existing bound of .
Paper analyzes convergence of entropy-regularized reinforcement learning methods.
problem Ensuring sub-optimality control in entropy-regularized RL.
method Unified analysis of regularized MPI and VI schemes, providing convergence rates.
result Established sufficient conditions for convergence and provided explicit rates.
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low d…
Adaptive optimal control using value iteration (VI) initiated from a stabilizing policy is theoretically analyzed in various aspects including the continuity of the result, the stability of the system operated using any single/constant resulting control policy, the stability of the system operated using the evolving/ti…
Tackling large approximate dynamic programming or reinforcement learning problems requires methods that can exploit regularities, or intrinsic structure, of the problem in hand. Most current methods are geared towards exploiting the regularities of either the value function or the policy. We introduce a general classif…
Efficiently compress pretrained models using RSI for improved predictive accuracy.
problem Efficiently compressing large pretrained models for practical deployment.
method Randomized subspace iteration (RSI) for low-rank approximation of pretrained models.
result RSI achieves near-optimal approximation quality and outperforms RSVD in predictive accuracy.
In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is u…
In this paper, we propose a new threshold-kernel jump-detection method for jump-diffusion processes, which iteratively applies thresholding and kernel methods in an approximately optimal way to achieve improved finite-sample performance. We use the expected number of jump misclassifications as the objective function to…
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation erro…
This work provides safety guarantees for iterative GP predictions.
problem Analytical intractability of uncertainty tracking in iterative GP predictions.
method Deriving formal probability error bounds for iterative GP predictions.
result Formal bounds ensure that GP trajectories lie within specified regions with high probability.
Improved API to achieve optimal error bound and query complexity in local planning.
problem Efficient local planning in discounted MDPs with linear approximation.
method Confident Approximate Policy Iteration (CAPI) for stationary policies, applying to local access simulators.
result Achieves optimal accuracy and query complexity bounds, improving over API.
The paper explores how duality applies to reinforcement learning.
problem Exploring the use of duality in reinforcement learning.
method Demonstrates duality in various RL methods, including value iteration and approximations.
result Lagrangian duality is found to be prevalent in reinforcement learning.
A new method calibrates value predictions in offline RL to improve reliability.
problem Difficulty in long-horizon value prediction in offline reinforcement learning.
method Bellman calibration, a weak reliability criterion, and Iterated Bellman Calibration.
result Finite-sample guarantees show that Bellman calibration error is controlled at nonparametric rates.
Improved Politex algorithm reduces regret bound to O(√T) with experience replay.
problem Learning in infinite-horizon MDPs with function approximation.
method Sharpened regret analysis of Politex algorithm, experience replay implementation.
result First high-probability O(√T) regret bound for computationally efficient algorithm.
Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squaresmath.OC
We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…
Method solves complex optimization problems with high probability bounds.
problem Nonlinear equality constrained stochastic optimization problems.
method Step-search sequential quadratic programming method.
result High-probability bound on iteration complexity for first-order stationarity.
Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as ne…
We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…
Recursive stochastic algorithms have gained significant attention in the recent past due to data driven applications. Examples include stochastic gradient descent for solving large-scale optimization problems and empirical dynamic programming algorithms for solving Markov decision problems. These recursive stochastic a…
We consider the networked multi-agent reinforcement learning (MARL) problem in a fully decentralized setting, where agents learn to coordinate to achieve the joint success. This problem is widely encountered in many areas including traffic control, distributed control, and smart grids. We assume that the reward functio…
Develops fast approximations for conditional Shapley values in linear and polynomial models.
problem Estimating conditional Shapley values using regression models is computationally expensive.
method A new approximative estimation method for conditional Shapley values using linear and polynomial regression models.
result Our method significantly reduces computation time compared to existing methods.
A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergencecs.GT
Policy gradient method proves convergence in imperfect-information games.
problem Policy gradient methods in imperfect-information games (EFGs).
method Policy gradient approach with best-iterate convergence.
result Policy gradient leads to provable best-iterate convergence in self-play EFGs.
Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcs.LG
New algorithm learns value and advantage functions for continuous-time Markov processes without structural assumptions.
problem Learning value and advantage functions for continuous-time Markov processes without structural assumptions.
method Proposes Sobolev-prox fitted -learning algorithm based on Hilbert-space positive definiteness and boundedness properties of Bellman operators.
result Identifies ellipticity as a key structural property enabling reinforcement learning for Markov diffusions.
Unified framework for AMP iterations using graph indexing.
problem Complex high-dimensional statistical inference problems.
method Graph-based indexing of AMP iterations, modular proof of state evolution.
result Unified SE equations for AMP iterations indexed by graphs.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
New algorithm learns optimal policies without explicit rewards.
problem Learning optimal policies without explicit rewards.
method Developed a computationally tractable algorithm for reward-free navigation using linear value iteration.
result PAC guarantees on learning near optimal value functions and policies.