LHIEM model predicts health, income, and employment over years.
problem Lack of path dependency in health policy simulations.
method Discrete-time microsimulation with Markov chain modules.
result Validates health care financing proposal through detailed modeling.
This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is t…
Algorithm optimizes lockdown policies balancing health and economy.
problem Balancing health and economic impacts of lockdowns during pandemics.
method Reinforcement learning to automatically compute lockdown policies.
result Algorithm learns optimal lockdown policies from disease and population data.
IntelligentPooling learns personalized mHealth policies from limited data.
problem Challenges in personalizing mHealth policies with limited user data and noise.
method Adaptive use of other users' data to learn personalized policies.
result Achieves 26% lower regret compared to state-of-the-art approaches.
Paper develops methods for evaluating mHealth interventions using historical data.
problem Evaluating the long-term effectiveness of mHealth interventions designed for near-term outcomes.
method Develops off-policy estimation techniques to infer long-term average outcomes from historical data.
result Provides estimators and confidence intervals for evaluating mHealth policies.
This note fills the gap in market-consistent valuation of lifelong health insurance products.
problem Market-consistent valuation of lifelong health insurance products is not well-addressed.
method Constructs a valuation portfolio to separate Best Estimate into policy data and financial instrument prices.
result The Best Estimate valuation is not uniquely determined by prevailing term structures and requires a stochastic model.
Develops a method to efficiently use offline data for RL policy optimization.
problem Lack of online data for offline RL in mobile health applications.
method Advantage learning framework using optimal Q-estimators.
result New policy converges faster than existing methods.
Simulation-based inference aids in predicting disease dynamics for health policy.
problem Predicting disease dynamics to inform public health interventions.
method Simulation-based inference using machine learning.
result Potential for efficient interpretable Bayesian inference in epidemiological models.
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.
Mobile apps and machine learning improve malaria prevention and treatment.
problem High malaria cases and deaths in low-income countries.
method Adaptive interventions using mobile health apps and machine learning.
result Increased malaria testing, adherence, and provider skills.
AACE learns treatment policies from EHRs using annotations to improve accuracy.
problem Learning treatment policies from multimodal EHRs with bias and inefficiency.
method Annotation-assisted coarsened effects (AACE) method.
result AACE outperforms existing methods in predicting treatment benefit from multimodal EHRs.
The paper develops methods to estimate value functions in reinforcement learning for infinite horizon problems.
problem Constructing confidence intervals for value functions in infinite horizon reinforcement learning.
method Modeling the Q-function using series/sieve method and recursively updating the policy using SAVE method.
result The proposed CI achieves nominal coverage even when the optimal policy is not unique.
EHR-MPC optimizes sepsis treatment using digital twins and inference-time control.
problem Optimal sepsis treatment policies are contested and difficult to adapt during inference.
method EHR-MPC decouples learning patient dynamics from treatment optimization, enabling inference-time control over learned digital twins.
result EHR-MPC achieves comparable off-policy performance and improved simulation performance compared to RL baselines.
Study optimizes health incentives to balance efficiency and fairness.
problem Designing health incentives to balance efficiency and fairness.
method Inverse behavioral optimization framework integrating QALY-based incentives and adaptive learning.
result Modern health systems operate near an efficiency-saturated frontier, with small fairness adjustments yielding diminishing returns.
The opioid epidemic in the United States claims over 40,000 lives per year, and it is estimated that well over two million Americans have an opioid use disorder. Over-prescription and misuse of prescription opioids play an important role in the epidemic. Individuals who are prescribed opioids, and who are diagnosed wit…
Although there are millions of transgender people in the world, a lack of information exists about their health issues. This issue has consequences for the medical field, which only has a nascent understanding of how to identify and meet this population's health-related needs. Social media sites like Twitter provide ne…
Study user engagement in mobile health apps for health workers in resource-poor settings.
problem Detect churn and tailor content for health workers in mobile health apps.
method Probabilistic and survival analysis of behavioral logs.
result Personalized measures of meaningful engagement can enhance health outcomes.
Efficient RL method uses SSL for small labeled data in health outcomes.
problem Lack of precise health outcome data for reinforcement learning.
method Semi-supervised learning approach for Q-learning and value estimation.
result Method efficiently estimates Q-function and value function, robust to mis-specification.
The distribution of health care payments to insurance plans has substantial consequences for social policy. Risk adjustment formulas predict spending in health insurance markets in order to provide fair benefits and health care coverage for all enrollees, regardless of their health status. Unfortunately, current risk a…
BREMEN optimizes policies offline with fewer data, achieving efficient deployment.
problem High cost of updating policies in real-world applications.
method Behavior-Regularized Model-ENsemble (BREMEN) algorithm for offline optimization.
result BREMEN achieves impressive deployment efficiency with 5-10 deployments, outperforming standard RL methods.
A lack of information exists about the health issues of lesbian, gay, bisexual, transgender, and queer (LGBTQ) people who are often excluded from national demographic assessments, health studies, and clinical trials. As a result, medical experts and researchers lack a holistic understanding of the health disparities fa…
Proposes a deep spectral Q-learning for mobile health data.
problem Personalized treatment assignment for patients with time-varying covariates.
method Integrates PCA with deep Q-learning for mixed frequency data.
result Mean return converges to optimal under estimated optimal policy.
Life-expectancy is a complex outcome driven by genetic, socio-demographic, environmental and geographic factors. Increasing socio-economic and health disparities in the United States are propagating the longevity-gap, making it a cause for concern. Earlier studies have probed individual factors but an integrated pictur…
Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effect…
RoME optimizes mobile health interventions by modeling user and time-specific effects.
problem Challenges in optimizing mobile health interventions due to participant heterogeneity, nonstationarity, and nonlinear relationships.
method RoME uses a Robust Mixed-Effects contextual bandit algorithm with random effects, network cohesion penalties, and debiased machine learning.
result RoME achieves robust regret bounds even with complex baseline rewards, demonstrating superior performance in simulations and studies.
Framework optimizes battery storage for markets by separating long-term degradation from short-term market dynamics.
problem Intractable computation due to timescale mismatch between battery degradation and market dynamics.
method Approximate dynamic programming with value function approximation and pseudo-time encoding.
result Policy outperforms benchmarks in real-time market scenarios.
Proposes pT-Learning for optimal dynamic treatment regimes in mHealth.
problem Challenges in learning optimal dynamic treatment regimes with large intervention options and infinite time horizon.
method Proximal Temporal consistency Learning (pT-Learning) framework for adaptively adjusting between deterministic and stochastic policies.
result Minimax estimator avoids double sampling issue and can incorporate off-policy data.
Paper uses RL to optimize daily step distribution for better health biomarkers.
problem Lack of personalized PA distribution recommendations for health biomarkers.
method Developed an offline reinforcement learning algorithm to learn optimal PA distributions.
result Learned optimal policy suggests more consistent daily steps and tailored recommendations.
Imagine a patient in critical condition. What and when should be measured to forecast detrimental events, especially under the budget constraints? We answer this question by deep reinforcement learning (RL) that jointly minimizes the measurement cost and maximizes predictive gain, by scheduling strategically-timed meas…
Traditional reinforcement learning agents learn from experience, past or present, gained through interaction with their environment. Our approach synthesizes experience, without requiring an agent to interact with their environment, by asking the policy directly "Are there situations X, Y, and Z, such that in these sit…
Study optimizes inspection and monitoring of deteriorating structures using POMDPs.
problem Optimizing decision-making for deteriorating engineering systems with uncertain observations.
method Developed theoretical and computational foundations for Value of Information (VoI) and Value of Structural Health Monitoring (VoSHM) in POMDPs.
result POMDP policies inherently leverage VoI to guide optimal observational actions, improving long-term cost.
BFTS uses Bayesian Additive Regression Trees for improved personalized mobile health interventions.
problem Adapting to complex, non-linear user behaviors in personalized mobile health interventions.
method Bayesian Forest Thompson Sampling (BFTS) integrates Bayesian Additive Regression Trees (BART) into the exploration loop of contextual bandits.
result BFTS achieves state-of-the-art regret on tabular benchmarks and improves engagement rates by over 30% in a behavioral intervention study.
We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed ex…
New bandit model for healthcare intervention planning.
problem Maximizing patient health with limited monitoring resources.
method Developed Collapsing Bandits model and derived optimal policies.
result 3-order-of-magnitude speedup in algorithm performance.
ESOP uses Bayesian optimization to find optimal lock-down schedules.
problem Finding optimal lock-down schedules balancing health and economy.
method Bayesian optimization interacting with epidemiological models.
result ESOP schedules balance public health and economic impacts.
Developed a neural topic model for classifying COVID-19 disinformation.
problem Tackles the challenge of disinformation during the COVID-19 pandemic.
method Classification-aware neural topic model (CANTM) for COVID-19 disinformation.
result Demonstrated the effectiveness of CANTM in classifying COVID-19 disinformation.
Excessively changing policies in many real world scenarios is difficult, unethical, or expensive. After all, doctor guidelines, tax codes, and price lists can only be reprinted so often. We may thus want to only change a policy when it is probable that the change is beneficial. In cases that a policy is a threshold on …
Paper introduces effect-invariance for better policy generalization.
problem Adapting policies to unseen environments efficiently.
method Introduces effect-invariance, a relaxation of full invariance, and develops testing procedures to test e-invariance directly from data.
result Effect-invariance enables zero-shot and few-shot policy generalization without assuming a causal graph.
Paper enhances RL policies using trust region optimization for offline data.
problem Limited data in offline RL settings.
method Trust region optimization for value enhancement.
result Enhanced policy values with faster convergence.
Risk prediction is central to both clinical medicine and public health. While many machine learning models have been developed to predict mortality, they are rarely applied in the clinical literature, where classification tasks typically rely on logistic regression. One reason for this is that existing machine learning…
Bayesian model for cost-effectiveness analysis with subgroup discovery.
problem Statistical challenges in cost-effectiveness analysis, especially with non-random treatment assignment and censored data.
method Developed a nonparametric Bayesian model using Dirichlet and Gamma processes to estimate cost-survival distributions and identify cost-effectiveness subgroups.
result Identified and estimated policy-relevant causal CEA estimands using a Bayesian nonparametric g-computation procedure.
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electroni…
We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is eval…
Agent decides when to measure latent states in RL to improve efficiency.
problem Costly state measurement in RL negatively affects future outcomes.
method Introduces AOMDP with measurement action, uses online RL and sequential Monte Carlo.
result Reduced uncertainty improves sample efficiency and policy value.
This paper develops explainable treatment policies for RPM using clinical knowledge.
problem Barriers to adoption of DHIs and lack of interpretability in purely black-box algorithms.
method Developed a pipeline for learning explainable treatment policies using clinician-informed representations.
result Policies learned from clinician-informed representations are more efficacious and efficient than black-box policies.
The ability to perform offline A/B-testing and off-policy learning using logged contextual bandit feedback is highly desirable in a broad range of applications, including recommender systems, search engines, ad placement, and personalized health care. Both offline A/B-testing and off-policy learning require a counterfa…
Bayesian networks learn sub-population differences from data.
problem Inference from a single network structure can be misleading when data populations are heterogeneous.
method A mixture of Bayesian networks where component probabilities depend on individual characteristics.
result Identifies both network structures and demographic predictors of sub-population membership.