Graph-based state representation improves deep RL performance.
problem High sample-complexity and starting with a good input representation in deep RL.
method Exploiting the graph structure of MDPs for effective state representation learning.
result Graph-based node representation methods outperform matrix-based methods in grid-world navigation tasks.
A new method for multi-agent planning on graphs outperforms existing approaches.
problem Planning coordination among multiple interacting agents on a graph.
method Variational perturbation theory applied to inference in large networks.
result Our method outperforms state-of-the-art methods in non-local cost function scenarios.
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved O ( H 3 / 2 d K ) O(H^{3/2}d\sqrt{K}) O ( H 3/2 d K ) regret bound for Exo-MDPs, matching lower bounds. In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…
Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved Ω ( ( H 3 S A / ε 2 ) log ( 1 / δ ) ) Ω((H^3SA/ε^2)\log(1/δ)) Ω (( H 3 S A / ε 2 ) log ( 1/ δ )) sample complexity lower bound. We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …
New algorithm improves graph-based active learning by identifying unexplored regions.
problem Improving graph-based active learning by identifying unexplored regions.
method Poisson Reweighted Laplacian Uncertainty Sampling (PWLL) with a diagonal perturbation.
result PWLL effectively identifies unexplored regions in graph-based data.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
A new method for CMDP solving without compromising safety constraints.
problem Solving CMDP problems while adhering to safety constraints.
method Decomposition into reconnaissance and planning MDPs.
result Achieves safe policies for any safety constraint set.
Bayesian analysis shows unlabeled data improve graph-based semi-supervised learning.
problem Improving semi-supervised learning with limited labeled data.
method Bayesian nonparametric approach using unlabeled data for graph-based learning.
result Posterior contracts optimally around the truth with sufficient unlabeled data.
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Reward suffices for convex MDPs, expanding RL to new problems.
problem Capturing goals as convex functions of stationary distribution.
method Reformulated as a min-max game using Fenchel duality.
result Convex MDPs require non-stationary reward functions.
End-to-end graph-based SSL learns all graph factors dynamically.
problem Learning quality of graph in SSL is crucial but difficult.
method Proposes an end-to-end approach to optimize all graph factors.
result Demonstrates effectiveness on benchmark datasets.
Graph-based methods for anomaly detection and semi-supervised learning.
problem Detecting unusual clinical actions and anomalies in hospital data.
method Label propagation, harmonic solution, regularization, graph connectivity analysis.
result Effective anomaly detection and semi-supervised learning methods for healthcare data.
The paper addresses statistical estimation in MDPs with confounders using instrumental variables.
problem Statistical estimation of value functions in MDPs with unobservable confounders.
method Two-stage estimator based on instrumental variables for confounded linear MDPs.
result Established statistical properties of the two-stage estimator, including error bounds and asymptotic normality.
Study on regret minimization in deterministic MDPs.
problem Minimizing regret in deterministic reinforcement learning.
method Logarithmic regret lower bounds, leveraging graph theory and cycles.
result Explicitly quantifies the fundamental limit of performance achievable by any learning algorithm.
Improves graph-based active learning for non-Gaussian models.
problem Efficiently selecting data points for labeling in graph-based semi-supervised learning.
method Approximates non-Gaussian distributions, introduces rank-one update and model change acquisition function.
result Enhanced active learning for graph-based SSL under non-Gaussian models.
Improved regret bound for MNL MDPs with variance-aware approach.
problem Optimal reinforcement learning for MNL MDPs with structured variance.
method Introducing a problem-dependent constant measuring average variance, proposing an algorithm with improved regret bound.
result Minimax optimal regret bound of O ( d H 2 σ ˉ T T ) O(dH^2\barσ_T\sqrt{T}) O ( d H 2 σ ˉ T T ) for structured MDPs. This work uses action equivariance to learn structured latent spaces for reinforcement learning.
problem Learning structured latent spaces for reinforcement learning.
method Introduced a contrastive loss function to enforce action equivariance on learned representations.
result Optimal policies in the abstract MDP can be successfully lifted to the original MDP.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
Galerkin method outperforms graph-based methods in spectral decompositions.
problem Improving spectral decomposition methods in machine learning.
method Restricting study to a small set of test functions using the Galerkin method.
result Statistical and computational superiority of Galerkin method over graph-based approaches.
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
It has recently been shown that if feedback effects of decisions are ignored, then imposing fairness constraints such as demographic parity or equality of opportunity can actually exacerbate unfairness. We propose to address this challenge by modeling feedback effects as Markov decision processes (MDPs). First, we prop…
The paper introduces MDP homomorphic networks for faster reinforcement learning.
problem Current reinforcement learning approaches do not exploit symmetries in the joint state-action space.
method Equivariant neural networks with group-structured symmetries (reflections, rotations).
result MDP homomorphic networks converge faster than unstructured baselines on various tasks.
UCBVI-γ algorithm minimizes regret in discounted MDPs.
problem Minimizing regret in discounted MDPs.
method Optimism in the face of uncertainty principle and Bernstein-type bonus.
result UCBVI-γ achieves nearly minimax optimal regret.
Survey of methods to incorporate external knowledge into stock price prediction.
problem Challenges in predicting stock prices due to market volatility and non-linearity.
method Survey of methods for acquiring and incorporating external knowledge into stock price prediction models.
result Systematic synthesis of previous studies on external knowledge types and their application in stock price prediction.
We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses. SCAL + ^+ + is a variant of SCAL (Fruit et al., 2018) that performs efficient exploration-exploitation in any unknown weakly-communicating MDP for which an upper bo…
New RL approach tackles non-linear MDPs without linear assumptions.
problem Sample efficiency in RL for complex, nonlinear MDPs with continuous states.
method Introduces EPW condition to relax linear structure requirements; provides sample-efficient RL algorithm.
result EPW condition allows solving MDPs without linear assumptions, including Atari games.
This paper introduces a novel graph signal processing framework for building graph-based models from classes of filtered signals. In our framework, graph-based modeling is formulated as a graph system identification problem, where the goal is to learn a weighted graph (a graph Laplacian matrix) and a graph-based filter…
New method for estimating and optimizing MDPs without stationarity.
problem Challenges in offline contextual MDP estimation without stationarity.
method Introduces a new adaptive estimation and cost optimization approach for contextual MDPs.
result First robust, theoretically backed method for offline contextual MDP estimation.
Graph-based weather prediction adapted for local models.
problem Applying neural weather prediction to limited area modeling.
method Adapting graph-based Neural Weather Prediction approach to local models.
result Validation of multi-scale hierarchical model extension for Nordic region.
Unified framework for solving MDPs with stochastic mirror descent.
problem Approximately solving infinite-horizon Markov decision processes (MDPs).
method Primal-dual stochastic mirror descent for MDPs with a unified framework.
result Computes ε-optimal policies with expected samples for both average-reward and discounted MDPs.
Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a i l d e O ( d 3 / 4 H 2 K 3 / 4 ) ilde{\mathcal{O}}(d^{3/4}H^2K^{3/4}) i l d e O ( d 3/4 H 2 K 3/4 ) regret bound. ASE safely explores unknown MDPs with unknown dynamics, improving sample efficiency.
problem Balancing exploration and safety in unknown MDPs with stochastic dynamics.
method Exploits analogies between state-action pairs to safely learn near-optimal policies.
result Empirically improves sample efficiency compared to existing methods.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
Paper proposes an efficient RL algorithm for discounted MDPs using feature mapping.
problem Efficient reinforcement learning for large state and action spaces.
method Uses feature mapping to represent states and actions in a low-dimensional space, proposing a novel algorithm with polynomial regret bound.
result Achieves a O ( d T / ( 1 − γ ) 2 ) O(d\sqrt{T}/(1-γ)^2) O ( d T / ( 1 − γ ) 2 ) regret bound, near-optimal up to a ( 1 − γ ) − 0.5 (1-γ)^{-0.5} ( 1 − γ ) − 0.5 factor. DARLING tackles non-stationary RL with guarantees, improving dynamic regret.
problem Non-stationary reinforcement learning in unknown change points.
method Detection Augmented Reinforcement Learning (DARLING) for tabular and linear MDPs.
result DARLING matches minimax lower bounds in tabular and linear MDPs.
Paper proves exponential lower bounds for policy iteration in MDPs.
problem Proving lower bounds for policy iteration in multi-action MDPs.
method Generalized previous results to k-action MDPs, constructed families of MDPs.
result Proved novel exponential lower bound of (3+k)2^(N/2-3) iterations.
Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision making under uncertainty. The classical approaches for solving MDPs are well known and have been widely studied, some of which rely on approximation techniques to solve MDPs with large state space and/or action space. However…