arXiv research

New formalism for decision making combines causal structures with MDPs, improving reinforcement learning performance.

The paper tackles finding optimal treatment sequences in continuous state spaces.

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…

Develops methods for finding counterfactual explanations in sequential decision making.

This manuscript contributes a general and practical framework for casting a Markov process model of a system at equilibrium as a structural causal model, and carrying out counterfactual inference. Markov processes mathematically describe the mechanisms in the system, and predict the system's equilibrium behavior upon i…

Enhances RL in partially observable, noisy environments by uncovering causal states.

Develops RL algorithm for non-Markovian, non-stationary reward streams.

A new meta-learner improves prediction of individualized outcomes in sequential decisions.

Dynamical-VAE learns causal dynamics from POMDPs using future information.

New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.

Intelligent agents can cope with sensory-rich environments by learning task-agnostic state abstractions. In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP). Our m…

Predictive rate-distortion analysis suffers from the curse of dimensionality: clustering arbitrarily long pasts to retain information about arbitrarily long futures requires resources that typically grow exponentially with length. The challenge is compounded for infinite-order Markov processes, since conditioning on fi…

PRL improves off-policy evaluation in partially observed MDPs.

Paper presents an algorithm for optimal regret in communicating Markov decision processes.

A new algorithm uses IVs to learn optimal policies from observational data.

In many settings (e.g., robotics) demonstrations provide a natural way to specify tasks; however, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the tasks, such as rewards or policies, can be safely composed and/or do not explicitly capture history dependen…

Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.

Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…

The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an…

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

New methods for Markov Blanket discovery using MML outperform existing approaches.

This work addresses the following question: Under what assumptions on the data generating process can one infer the causal graph from the joint distribution? The approach taken by conditional independence-based causal discovery methods is based on two assumptions: the Markov condition and faithfulness. It has been show…

Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.

CPP improves predictive model outputs for better intervention decisions.

Method infers causal structure from system behaviors using RKHS and kernel $ε$-machines.

Extends expected value framework for cost-sensitive causal decision-making.

Study evaluates policies in partially observable environments without full model specification.

Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.

This research tackles intervention-centric causal reasoning in learning agents by using meta-learning.

Risk measures applied to dynamic Markov processes with varying risk aversion.

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…

Bayesian method infers local rules for collective animal movement.

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual traject…

The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.

We establish causal semantics for SDEs and develop methods to reason about them.

New algorithm identifies best policy in MDPs faster.

The paper provides PAC bounds for estimating causal effects using covariate adjustment with a valid set.

Transformers learn causal structure through gradient descent on self-attention mechanisms.

We develop robust Markov Decision Processes with risk measures for uncertain environments.

Q-learning for average cost MDPs gets a concentration bound.

New Q-learning method achieves optimal sample complexity for average-reward problems.

We introduce Threatened Markov Decision Processes (TMDPs) as an extension of the classical Markov Decision Process framework for Reinforcement Learning (RL). TMDPs allow suporting a decision maker against potential opponents in a RL context. We also propose a level-k thinking scheme resulting in a novel learning approa…

Improves DRL for long-term causal inference with semiparametric methods.

This paper analyzes risk-sensitive reinforcement learning with Conditional Value-at-Risk (CVaR) for robust Markov Decision Processes.

We characterize value functions in partially observable MDPs as semi-algebraic sets.

New RL framework models continuous-time dynamics using neural ODEs.

Study minimax-optimal rates for offline decision-making with function approximation.

New algorithm learns causal structures by intersecting Markov blankets.