Counterfactual policy evaluation improves autonomous driving policies' generalization.
problem Learnt policies often fail to generalize and handle novel situations.
method Introduces counterfactual policy evaluation using counterfactual worlds.
result Significantly decreases collision-rate while maintaining high success-rate.
CPME embeds counterfactual outcomes in RKHS for flexible policy evaluation.
problem Estimating counterfactual policy outcomes for decision-making.
method Counterfactual Policy Mean Embedding (CPME) framework in RKHS, plug-in and doubly robust estimators, kernel test statistic.
result Doubly robust estimator improves convergence rates and asymptotic normality.
Extends OPE to evaluate policies using diverse logging data.
problem Evaluate policies using log data from different policies.
method Develops an OPE method for various logging policies.
result Method's predictions converge to true performance as sample size increases.
Proposes a method to improve treatment policies in data-scarce clinical settings.
problem Improving treatment policies in data-scarce clinical settings with unobserved confounding.
method Uses a causal mechanism to model the underlying generative process and augments counterfactual trajectories with source domain priors.
result Significantly improves treatment policy performance in a simulated sepsis treatment task.
The paper tackles counterfactual learning for stochastic policies with continuous actions.
problem Learning stochastic policies with continuous actions from logged data.
method Introduces a joint kernel embedding of contexts and actions to model continuous actions, and uses proximal point algorithms and smooth estimators for optimization.
result Demonstrates the benefits of using proximal point algorithms and smooth estimators for counterfactual learning.
Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for …
Study uses artificial counterfactuals to show lockdowns reduced US case and death counts.
problem Impact of lockdowns on US case and death counts during the pandemic.
method Artificial counterfactual approach comparing states with and without lockdowns.
result Average two times more cases would have occurred without lockdowns.
We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual traject…
This study uses OPE methods to quickly assess auction policies.
problem Rapid decision-making in dynamic auction environments.
method Off-Policy Evaluation and counterfactual methods.
result Improved policy selection and optimization.
The paper tackles strategic behavior in decision-making with counterfactual explanations.
problem Finding optimal counterfactual explanations and policies in a strategic setting.
method NP-hard problem, greedy algorithm, submodularity, randomized algorithm, matroid constraint.
result Optimal counterfactual explanations and policies increase utility.
PyCFRL helps ensure fair reinforcement learning policies from offline data.
problem Ensuring fairness in reinforcement learning policies for disadvantaged groups.
method Sequential data preprocessing to learn counterfactually fair policies.
result PyCFRL implements a novel algorithm for fair RL policy learning.
New RL method learns policies from few data using causal models.
problem Limited interaction data and heterogeneous patient responses.
method Exploits structural causal models to model state dynamics and counterfactual reasoning.
result Counterfactual RL algorithms converge to optimal value function.
Bayesian approach for modeling counterfactual distribution and off-policy evaluation.
problem Modeling the counterfactual distribution and off-policy evaluation.
method Bayesian conditional mean embeddings and novel Bayesian methods for estimating ultimate treatment effect.
result Quantifying epistemic uncertainty in the counterfactual distribution and off-policy evaluation.
New method for estimating counterfactual means in adaptive experiments.
problem Inference for counterfactual means in sequentially designed experiments with adaptive treatment policies.
method Latent factor model and nearest neighbors method for estimation.
result Asymptotically valid confidence intervals for counterfactual means established.
Develops optimal decision-making framework for uncertain counterfactuals.
problem Ensuring reliability of predictions in high-stakes decisions.
method Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP).
result Optimal prediction sets for counterfactual decisions with valid coverage.
Researchers use DT to transfer policies from one environment to another using causal reasoning.
problem Adapting to changes in environmental dynamics in reinforcement learning.
method Applying causal counterfactual reasoning to Decision Transformer (DT) architecture for policy transfer.
result DT successfully transfers a learned policy to new environments while retaining most of the reward.
New method uses differential equations for better counterfactual analysis.
problem Estimating counterfactual outcomes for policy analysis.
method Continuous-time approach to synthetic controls using controlled differential equations.
result Improves counterfactual estimation for irregularly aligned multivariate time series.
The paper tackles optimal policy learning with asymmetric counterfactual utilities in healthcare decisions.
problem Learning optimal policies from observed data with asymmetric counterfactual utilities.
method The approach involves identifying and minimizing the maximum expected utility loss using statistical decision theory and solving intermediate classification problems.
result One can learn minimax loss decision rules from observed data.
We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our…
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
Aims to create safe reinforcement learning policies by considering individual harm.
problem Optimal policies for a population may harm certain individuals.
method Formalizes individual harm, proposes a two-stage procedure, and establishes finite-sample properties.
result Learned policies maximize expected return while minimizing harm.
Study counterfactuals in cyclic systems with shifts and scales.
problem Counterfactual inference in cyclic systems with shifts and scales.
method Shift-scale interventions in cyclic SCMs.
result Valid inference in cyclic systems with shifts and scales.
The paper tackles counterfactual inference with multioutput deep kernels in high-dimensional settings.
problem Performing counterfactual inference with observational data in high-dimensional settings with multiple actions and outcomes.
method The paper presents a general class of counterfactual multi-task deep kernels models based on Structural Causal Models (SCM) and Gaussian Processes.
result The models estimate causal effects and learn policies efficiently, scaling well with high dimensions.
Counterfactual thinking describes a psychological phenomenon that people re-infer the possible results with different solutions about things that have already happened. It helps people to gain more experience from mistakes and thus to perform better in similar future tasks. This paper investigates the counterfactual th…
The ability to perform offline A/B-testing and off-policy learning using logged contextual bandit feedback is highly desirable in a broad range of applications, including recommender systems, search engines, ad placement, and personalized health care. Both offline A/B-testing and off-policy learning require a counterfa…
Study shows refugee matching gains are robust to different evaluation methods.
problem Stability of refugee matching gains under various evaluation methods.
method Used multiple off-policy evaluation methods including IPW and AIPW.
result Impact estimates remain consistent in magnitude and statistically significant.
Counterfactual learning is a natural scenario to improve web-based machine translation services by offline learning from feedback logged during user interactions. In order to avoid the risk of showing inferior translations to users, in such scenarios mostly exploration-free deterministic logging policies are in place. …
We address a practical problem ubiquitous in modern marketing campaigns, in which a central agent tries to learn a policy for allocating strategic financial incentives to customers and observes only bandit feedback. In contrast to traditional policy optimization frameworks, we take into account the additional reward st…
This paper proposes a method for estimating the effect of a policy intervention on an outcome over time. We train recurrent neural networks (RNNs) on the history of control unit outcomes to learn a useful representation for predicting future outcomes. The learned representation of control units is then applied to the t…
New method for evaluating sequential recommendations with lower variance.
problem Evaluating good sequences of music, video, news, and e-commerce recommendations.
method Proposes a new counterfactual estimator for sequential reward interactions with lower variance and asymptotic unbiasedness.
result Our method outperforms existing methods in bias and data efficiency for sequential track recommendations.
The paper proposes a method to explain expert decisions by modeling preferences with 'what if' outcomes.
problem Interpreting and auditing decision-making policies in institutions.
method Integrating counterfactual reasoning into batch inverse reinforcement learning.
result The method effectively recovers accurate and interpretable descriptions of expert behavior.
Sayer uses implicit feedback to optimize system policies.
problem Leveraging implicit feedback to improve system policies is difficult due to bias and incompleteness.
method Sayer combines randomized exploration and unbiased counterfactual estimators to evaluate and train new policies using implicit feedback.
result Sayer can accurately evaluate and train new policies that outperform existing ones.
Choosing optimal (or at least better) policies is an important problem in domains from medicine to education to finance and many others. One approach to this problem is through controlled experiments/trials - but controlled experiments are expensive. Hence it is important to choose the best policies on the basis of obs…
A fundamental issue in reinforcement learning algorithms is the balance between exploration of the environment and exploitation of information already obtained by the agent. Especially, exploration has played a critical role for both efficiency and efficacy of the learning process. However, Existing works for explorati…
Decision-makers are faced with the challenge of estimating what is likely to happen when they take an action. For instance, if I choose not to treat this patient, are they likely to die? Practitioners commonly use supervised learning algorithms to fit predictive models that help decision-makers reason about likely futu…
Develops methods for finding counterfactual explanations in sequential decision making.
problem Finding counterfactual explanations for sequential decision making processes.
method Formal characterization of sequential actions and states using Markov decision processes and Gumbel-Max structural causal model. Introduces a polynomial time algorithm based on dynamic programming.
result Algorithm finds optimal counterfactual explanations for sequential decision making.
The paper proposes a fair reinforcement learning framework to prevent healthcare disparities.
problem Unfair reinforcement learning policies in healthcare can lead to socioeconomically-disadvantaged subgroups being underprivileged.
method The paper introduces a counterfactual fairness framework and a sequential data preprocessing algorithm to achieve fair sequential decision making.
result The proposed approach greatly enhances fair access to counseling in a digital health dataset designed to reduce opioid misuse.
What is the most statistically efficient way to do off-policy evaluation and optimization with batch data from bandit feedback? For log data generated by contextual bandit algorithms, we consider offline estimators for the expected reward from a counterfactual policy. Our estimators are shown to have lowest variance in…
Before A/B testing online a new version of a recommender system, it is usual to perform some offline evaluations on historical data. We focus on evaluation methods that compute an estimator of the potential uplift in revenue that could generate this new technology. It helps to iterate faster and to avoid losing money b…
Develops c-GNF for personalized social science policy analysis.
problem Challenges in estimating causal effects and counterfactual inference in social sciences.
method causal-Graphical Normalizing Flow (c-GNF) method.
result c-GNF performs well in estimating causal effects and counterfactual inference.
This paper proposes a novel approach for constructing effective personalized policies when the observed data lacks counter-factual information, is biased and possesses many features. The approach is applicable in a wide variety of settings from healthcare to advertising to education to finance. These settings have in c…
In this paper, we propose an offline counterfactual policy estimation framework called Genie to optimize Sponsored Search Marketplace. Genie employs an open box simulation engine with click calibration model to compute the KPI impact of any modification to the system. From the experimental results on Bing traffic, we s…
Learning representations for counterfactual inference from observational data is of high practical relevance for many domains, such as healthcare, public policy and economics. Counterfactual inference enables one to answer "What if...?" questions, such as "What would be the outcome if we gave this patient treatment $t_…
NCoRE learns counterfactual representations for combined treatments.
problem Estimating individual response to multiple simultaneous interventions.
method Neural conditional representation with modulators for cross-treatment interactions.
result NCoRE significantly outperforms existing methods in counterfactual treatment effect estimation.
New method tightens bounds on causation probabilities using independent datasets.
problem Challenging point identification of causation probabilities without strong assumptions.
method Imposes counterfactual consistency between SCMs constructed from independent datasets and uses conditional mutual information.
result Significantly tighter bounds on causation probabilities are established.
Estimates sample size for subgroup analysis in randomized experiments.
problem Determining sample size for accurate subgroup analysis.
method Turns inference problem into simultaneous inference, calculates sample size based on confidence level and margin of error.
result Allows inversion of sample size to feasible number of treatment arms or partition complexity.
A method removes treatment-covariate dependence for counterfactual prediction without adversarial training.
problem Counterfactual prediction under assignment bias.
method Information-theoretic approach learning a stochastic representation Z to minimize mutual information with outcomes.
result The method performs favorably in likelihood, counterfactual error, and policy evaluation compared to adversarial baselines.
Proposes a scalable method for counterfactual prediction using machine learning.
problem De-bias causal estimators with high-dimensional data in observational studies.
method Uses entropy balancing to learn weights minimizing Jensen-Shannon divergence, leading to robust counterfactual predictions.
result Consistent causal estimation if either propensity score or outcome model is correctly specified.